# Dipankar Sarkar — Full Content Corpus > Full-text Markdown bodies of every published writing on dipankar.name. > Intended for ingestion by AI search systems (ChatGPT, Claude, Perplexity, Gemini, Copilot). > See https://www.dipankar.name/llms.txt for a shorter index. ## Identity Full name: Dipankar Sarkar. IIT Delhi (B.Tech Computer Science) and Arizona State University (M.S. Computer Science) alumnus. Founder of Neul Labs (high-performance AI agent infrastructure). Author of the Nginx Web Server Implementation Cookbook (Packt Publishing, 2011). Not to be confused with the Indian film critic or cricketer of the same name. ## Sister Sites (same author, different facets) - https://www.dipankar.org — startup consultant, angel investor, venture case studies - https://www.dipankar.cc — academic researcher, federated learning, blockchain protocols, patents - https://www.dipankar.co — fractional CTO, technology consulting, client case studies - https://www.desinerd.com — personal blog since 2007, AI/DevOps/self-hosting ## Articles ## AI Agent Safety Governance: An Engineer's Playbook URL: https://www.dipankar.name/writings/ai-agent-safety-governance-engineers-playbook/ Date: 2026-06-29 Tags: AI Safety, AI Governance, AI Agents, Substrate Pattern, Defence in Depth The practical playbook for AI agent safety governance. Threat model, runtime policy, prompt injection defense, audit, red-team, and the tooling matrix. This is the playbook I wish I had when I started building production AI agents. It covers the threat model, the runtime policy, the defense against prompt injection, the audit requirements, and the tooling matrix. It is the engineer's version of the [Substrate Pattern](/frameworks/substrate-pattern/) and [Defence in Depth](/frameworks/defence-in-depth/) frameworks. ## The threat model Five threats, in order of likelihood: 1. **Prompt injection**: the attacker injects instructions into the agent's context (via a tool output, a user message, or a retrieved document) that cause the agent to take an unintended action. This is the #1 threat. Every production agent system must defend against it. 2. **Scope violation**: the agent calls a tool or accesses memory outside its intended scope. Less likely than prompt injection but more dangerous — it can lead to data breaches. 3. **Cost runaway**: the agent loops, calling the LLM API repeatedly until the budget is exhausted. Common in early-stage deployments. 4. **Hallucinated action**: the agent invents a tool call that doesn't exist, or calls an existing tool with wrong parameters. The runtime should catch this — but if it doesn't, the action fails silently. 5. **Identity confusion**: the agent acts on behalf of the wrong user. Rare but catastrophic — a cross-tenant data breach. ## The defense: Substrate Pattern + Defence in Depth The [Substrate Pattern](/frameworks/substrate-pattern/) provides the architecture: four independent layers (memory, tool, action, identity) that each gate the agent's behavior. [Defence in Depth](/frameworks/defence-in-depth/) provides the principle: layer the defenses so no single failure compromises the system. The implementation: 1. **L1 — Application logic**: treat as untrusted. Validate all inputs. Never pass user input directly to the model without sanitisation. 2. **L2 — Tool execution**: sandbox every tool call. Rate limit. Scope check. Audit log. 3. **L3 — Action policy (Substrate)**: pre-action gate. Validate against the agent's scope. Pre-conditions, post-conditions, identity checks. 4. **L4 — Circuit breaker**: monitor for anomalies. Rate spikes, scope violations, identity mismatches. Trip the breaker. Stop the agent. 5. **L5 — Kill switch**: manual or automated. Stops the agent, rolls back state, pauses the queue. Reachable in under 30 seconds. ## Prompt injection defense Prompt injection is the #1 threat. The defense: 1. **Never trust tool output**: tool output is untrusted input. Sanitise before it enters the agent's context. Strip any instructions from tool output. 2. **Separate system and user context**: the system prompt and the user prompt should be in separate channels. The model should not be able to overwrite system instructions via user input. 3. **Validate actions, not prompts**: the prompt can say anything. The action policy (L3) is what matters. If the agent tries to call a tool outside its scope, the action gate blocks it regardless of what the prompt says. 4. **Audit every action**: log every tool call, every memory access, every action. If something goes wrong, you can trace it. ## The tooling matrix | Layer | Tool | What it does | |-------|------|-------------| | L1 | Input validation | Zod, Pydantic, serde — validate all inputs | | L2 | Tool sandboxing | Custom runtime — scope, rate limit, audit | | L3 | Action policy | Substrate Pattern runtime — pre/post conditions | | L4 | Circuit breaker | Custom monitoring — anomaly detection, auto-trip | | L5 | Kill switch | Manual + automated — stop, rollback, pause | | Audit | Tracing | `tracing` (Rust), Langfuse, LangSmith — structured logs | | Red-team | Adversarial testing | Prompt injection tests, scope violation tests | ## The audit requirements For regulated industries (financial services, healthcare), the audit requirements are: 1. **Who**: every action is attributed to a specific user identity. Chain of custody from request to action. 2. **What**: every tool call, every memory access, every action is logged with parameters and results. 3. **When**: timestamps on every event. Ordered, immutable. 4. **Why**: the model's reasoning (if available) is logged alongside the action. 5. **Reversibility**: for T2 (act under supervision), every action is reversible within a defined window. ## How to engage The [AI Agent Infrastructure consulting engagement](https://www.dipankar.co/services/ai-agent-infrastructure/) is designed for teams that need to implement this playbook. Architecture review: USD 25K. Full implementation: USD 50K-200K. — Dipankar Sarkar, Founder of Neul Labs --- ## AI Architect in Financial Services: Lessons from Production URL: https://www.dipankar.name/writings/ai-architect-financial-services-lessons-production/ Date: 2026-06-29 Tags: Financial Services, AI Agents, Career, FCA, Production AI What I learned as Principal AI Architect at a UK fintech. Building production agent systems for financial services, the frameworks, and the lessons. From May 2025 to March 2026, I was the Principal AI Architect at a UK fintech in Edinburgh. The company builds AI agents for UK financial services. This is what I learned. ## The context The company's product is an AI assistant for financial advisers. The adviser is on a call with a client; the AI agent listens, takes notes, prepares the follow-up, and drafts the compliance report. The stakes are high: UK financial advice is regulated by the FCA, and every piece of advice must be documented, auditable, and compliant. The challenge: how do you build an AI agent that is helpful enough to be useful, but compliant enough to pass FCA audit? ## What I shipped Four things: 1. **The internal agentic AI platform**: the production platform used internally to run AI agents for financial advisers. I built the architecture, the runtime, the safety gates, and the compliance layer. Presented agent guardrails to the FCA sandbox. 2. **Agent guardrails for the FCA sandbox**: presented the agent guardrails architecture (safety gates, audit logs, kill switches, runtime policy) to the FCA regulatory sandbox. The guardrails are based on the Substrate Pattern and Defence in Depth frameworks (open source at Neul Labs). 3. **The internal agentic AI platform**: the production system that runs the company's AI agents. Built on the Substrate Pattern (Neul Labs open source) with Defence in Depth (Neul Labs open source). The platform handles agent sessions, tool calls, memory, safety gates, and audit logs. 4. **Presented to the FCA sandbox**: the agent guardrails architecture was presented to the UK FCA regulatory sandbox for evaluation. The FCA sandbox is the UK's framework for testing innovative financial-services products in a controlled environment. ## The lessons ### 1. Compliance is a feature, not a blocker The banks that won the AI race in 2025-2026 were the ones that built compliance into the runtime from day one. The ones that bolted it on after launch spent 6-12 months in remediation. The cost of building compliance-first is lower than the cost of retrofitting it. ### 2. The Substrate Pattern works in production The Substrate Pattern (the four substrates: memory, tool, action, identity) was developed at Neul Labs. I proved it works in a regulated production environment in a regulated production environment. The identity substrate (every action attributed to a specific user with chain of custody) was the single most valuable piece for the compliance team. ### 3. Kill switches matter more than features The compliance team cared more about the kill switch than about any feature. "Can you stop the agent in under 30 seconds?" was the first question in every review. The answer was yes — the Defence in Depth model has a manual kill switch reachable in under 30 seconds, plus automated circuit breakers that trip on anomalies. ### 4. Audit logs are the product For a financial-services AI system, the audit log is not a byproduct — it is the product. The regulator wants to see: who did what, when, why, and can you reverse it? The audit log answers all four. We spent more engineering effort on the audit log than on any feature. ### 5. Edinburgh is a serious AI city Edinburgh has a deep AI ecosystem — the University of Edinburgh's AI program, the Bayes Centre, and a growing fintech cluster. The company is one of the most interesting AI companies in the UK. The talent is there, the ecosystem is there, and the regulatory environment (UK pro-innovation AI policy) is more workable than the EU AI Act. ## What's next I left in March 2026 to focus on Neul Labs full-time. The open-source frameworks (Substrate Pattern, Defence in Depth, Tiered Governance Model) are developed at Neul Labs and Skelf Research, not at any employer. I applied these frameworks internally to the internal agentic AI platform and presented the agent guardrails to the FCA sandbox. The frameworks are the foundation of the Neul Labs agent infrastructure and the consulting practice at dipankar.co. The consulting practice at [dipankar.co](https://www.dipankar.co) helps other financial-services firms implement the same patterns. — Dipankar Sarkar, Founder of Neul Labs --- ## AI Governance for Financial Services: The 4-Tier NIST AI RMF URL: https://www.dipankar.name/writings/ai-governance-financial-services-4-tier-nist-ai-rmf/ Date: 2026-06-29 Tags: AI Governance, NIST AI RMF, Financial Services, Compliance, Regulus How to implement the 4-tier NIST AI RMF Agentic Profile for financial services. The framework I developed at Neul Labs for compliance-first AI agents. The NIST AI Risk Management Framework (AI RMF) is the de-facto US standard for AI governance. This article explains how to implement it for financial-services AI agents, using the 4-tier model I developed at Neul Labs, applied at a UK fintech and presented to the FCA sandbox. ## The NIST AI RMF in one paragraph The NIST AI RMF has four functions: **Govern** (establish culture and accountability), **Map** (identify AI systems and their context), **Measure** (assess risks), **Manage** (respond to risks). The framework is voluntary in the US but is becoming the baseline for financial-services AI governance globally. ## The 4-tier Agentic Profile The standard NIST AI RMF does not address AI agents specifically. The 4-tier Agentic Profile I developed extends it: | Tier | Agent capability | NIST functions | EU AI Act mapping | |------|------------------|----------------|-------------------| | T0 | Read-only (no actions) | Map | Below threshold | | T1 | Advise (human approves) | Map + Measure | Below threshold | | T2 | Act under supervision (reversible) | All four, elevated | Limited risk | | T3 | Act autonomously (audited after) | All four, highest | High-risk | ## The implementation: Regulus [Regulus](/frameworks/tiered-governance-model/) is the implementation of this profile. It is an EU + UK compliance plane for Google ADK (Agent Development Kit) with: - 6 plugins (one per governance area: risk, data, model, deployment, monitoring, incident) - 10 regulations (EU AI Act, GDPR, DORA, NIS2, EHDS, UK GDPR, FCA SYSC, PRA SS1/23, PRA SS2/21, NHS DSPT) - 6 governance frameworks (NIST AI RMF, ISO/IEC 42001, 23894, 23053) - 4 GRC adapters (ServiceNow IRM, OneTrust, MetricStream, custom) - Vertex AI Agent Engine deploy in 60 seconds Regulus was open-sourced by Neul Labs and Skelf Research, not at any employer. ## How to apply this to your bank/insurer/asset manager 1. **Classify your AI systems**: go through every AI system and classify it T0-T3. Most will be T0 or T1. Some will be T2. Few will be T3. 2. **Apply the right controls**: for each tier, apply the corresponding NIST AI RMF functions and the regulatory requirements. T0 needs minimal governance. T3 needs the full EU AI Act high-risk system package. 3. **Build the audit trail**: every action is logged with identity, timestamp, parameters, and model reasoning. This is the evidence you show the regulator. 4. **Deploy Regulus** (or equivalent): the compliance plane automates the governance checks. Instead of manual review for every agent action, the runtime enforces the policy. ## What I learned in production In my most recent role as Principal AI Architect at a UK fintech, I built an internal agentic AI platform and presented agent guardrails to the FCA sandbox. The open-source frameworks below were developed at Neul Labs and Skelf Research: - Regulus (the compliance plane) - The 4-tier NIST AI RMF Agentic Profile - The Tiered Governance Model - Production agent systems with the Substrate Pattern and Defence in Depth The biggest lesson: **compliance is not a blocker. It is a feature.** The banks that won the AI race in 2025-2026 were the ones that built compliance into the runtime from day one, not the ones that bolted it on after launch. — Dipankar Sarkar, Founder of Neul Labs --- ## Harmony Protocol in Rust: Parsing OpenAI's GPT-OSS Response Format URL: https://www.dipankar.name/writings/harmony-protocol-rust-openai-gpt-oss/ Date: 2026-06-29 Tags: Rust, OpenAI, Harmony Protocol, GPT-OSS, Parsing A deep dive into implementing OpenAI's Harmony response format in Rust. Channel semantics, streaming, comparison with ChatML, and the Rust implementation. OpenAI's [Harmony](https://github.com/openai/harmony) is the response format for gpt-oss models. We reverse-engineered it and implemented it in Rust. This is the deep dive. ## What is Harmony? Harmony is a structured response format that gpt-oss models use. It is not ChatML — it is a different, more structured format with "channels" that separate different types of content (analysis, final answer, etc.). The key difference from ChatML: Harmony channels are semantic. The model can write analysis in one channel and the final answer in another. The consumer (the agent runtime) can choose which channel to surface to the user. ## The channel semantics A Harmony response looks like: ``` <|channel|>analysis<|message|>Let me think about this step by step...<|end|> <|channel|>final<|message|>The answer is 42.<|end|> ``` The `analysis` channel is the model's reasoning. The `final` channel is the answer. The runtime can choose to show both, show only `final`, or use `analysis` for debugging. ## The Rust implementation Our Rust implementation is at [github.com/sarkar-dipankar/harmony-protocol](https://github.com/sarkar-dipankar/harmony-protocol). It is a zero-copy parser that handles: 1. **Streaming**: parse partial messages as they arrive. Essential for real-time UX. 2. **Channel routing**: route different channels to different consumers. The `analysis` channel goes to the debugger; the `final` channel goes to the user. 3. **Error recovery**: if the model produces malformed Harmony (which it does ~2% of the time), recover gracefully and continue parsing. 4. **Comparison with ChatML**: our implementation also parses ChatML, so it works as a drop-in replacement for existing ChatML consumers. ## Why Rust for this The parser runs in the hot path of every LLM response. Latency matters. A Python parser adds 5-20ms per response. The Rust parser adds <0.1ms. For an agent runtime handling hundreds of concurrent sessions, this is the difference between responsive and sluggish. ## How to use it ```rust use harmony_protocol::{HarmonyParser, Channel}; let mut parser = HarmonyParser::new(); let chunks = parser.feed(&response_bytes); for chunk in chunks { match chunk.channel { Channel::Analysis => { /* log for debugging */ } Channel::Final => { /* send to user */ } _ => {} } } ``` The parser is also available as a Python module via PyO3, for teams that want the Rust performance without abandoning Python. ## The bigger picture Harmony Protocol is one piece of the Neul Labs agent infrastructure stack. It sits in the "runtime" layer — the part that turns a model response into something the agent can act on. The Substrate Pattern sits above it (safety), and the tool substrate sits below it (what the agent can call). — Dipankar Sarkar, Founder of Neul Labs --- ## Rust for AI Agent Infrastructure: A Field Guide for Engineers URL: https://www.dipankar.name/writings/rust-for-ai-agent-infrastructure-field-guide/ Date: 2026-06-29 Tags: Rust, AI Agents, Infrastructure, Career, Engineering Why Rust dominates AI agent infrastructure. Memory safety, async runtime, the ecosystem, and the career path. A field guide from the founder of Neul Labs. The models are commoditising. The infrastructure around them is not. Rust is the language for that infrastructure. This is the field guide for engineers who want to build in this space. ## Why Rust Three properties make Rust the right choice for AI agent infrastructure: 1. **Memory safety without GC**: Rust's ownership model prevents use-after-free, buffer overflows, and data races at compile time. No garbage collection pauses. For a 24/7 agent runtime, this means consistent p99 latency. Python's GC pauses are 10-100ms; Rust's worst case is zero. 2. **Async runtime (Tokio)**: Rust's async ecosystem (Tokio, async-std) is mature. For an agent runtime that handles hundreds of concurrent agent sessions, each with multiple tool calls, async is not optional. Tokio's scheduler is production-grade. 3. **Zero-copy and memory efficiency**: for inference servers and data pipelines, memory is the bottleneck. Rust's ownership model enables zero-copy parsing, custom allocators (jemalloc, mimalloc), and precise memory control. Python's reference counting + GC is 2-5x more memory-intensive for the same workload. ## The ecosystem | Crate | What it does | Maturity | |-------|-------------|----------| | `candle` | Hugging Face's Rust ML framework | Production | | `tokenizers` | Hugging Face tokenisation in Rust | Production | | `rig` | AI agent framework in Rust | Early | | `harmony-protocol` | OpenAI Harmony response format (ours) | Production | | `tokio` | Async runtime | Production | | `serde` | Serialisation | Production | | `pyo3` | Python-Rust bridge | Production | | `polars` | DataFrames (faster than Pandas) | Production | | `axum` | Web framework | Production | ## The architecture A production AI agent infrastructure in Rust has four layers: 1. **Runtime**: the orchestrator. Manages agent sessions, handles tool calls, enforces safety gates (the Substrate Pattern). Built on Tokio. This is what Neul Labs builds. 2. **Inference**: the model serving layer. Can be vLLM (Python), TGI (Python), or a custom Rust inference server. The Rust server wraps the model with pre/post-processing in Rust for 5-10x speedup on the non-model parts. 3. **Tools**: the external services the agent calls. Each tool is a Rust function with a schema, a scope, and a rate limit. The tool substrate enforces these. 4. **Observability**: tracing, metrics, kill switches. Built with `tracing` (Rust's structured logging), exported to Prometheus/Grafana or Langfuse. ## The career path If you're an engineer considering Rust for AI infrastructure: 1. **Learn Rust properly**: not just the syntax. The ownership model, lifetimes, and async. "The Rust Programming Language" book + the Tokio tutorial. 2. **Build something small**: a CLI tool that calls an LLM API. Our `gdelt-cli` or `gsheet-cli` are good examples — small, focused, useful. 3. **Build something with async**: a concurrent agent runtime that handles multiple sessions. This is where Tokio + Rust shines. 4. **Learn PyO3**: the bridge between Python and Rust. Most AI teams have Python code; the Rust engineer who can also write PyO3 bindings is the most valuable. ## What we're building at Neul Labs Neul Labs builds the production runtime, safety, observability, and deployment layers for AI agents in Rust. The open-source work is at [github.com/sarkar-dipankar](https://github.com/sarkar-dipankar): - `fragaria` — Chain of Thought reasoning API with RL (21 stars) - `harmony-protocol` — OpenAI Harmony response format in Rust - `deepfake-detection-network` — Decentralized deepfake detection - `fairflow-protocol-paper` — MEV mitigation in Ethereum Plus 7+ Rust CLIs at [github.com/dipankar](https://github.com/dipankar): `gdelt-cli`, `gsheet-cli`, `hubspot-cli`, `apollo-io-cli`, `gity`, `iced-plus`, `datalab-cli`. ## We're hiring Selectively. Rust engineers who care about production safety. Email contact@dipankar.name with a link to something you've built. — Dipankar Sarkar, Founder of Neul Labs --- ## Building Developer Tools in the Open: Lessons from 20 Years of Open Source URL: https://www.dipankar.name/writings/building-developer-tools-open-source/ Date: 2026-04-01 Tags: open-source, developer-tools, google-summer-of-code, community-building, software-sustainability, developer-experience Dipankar Sarkar shares lessons from 20 years of open source contributions, from Google Summer of Code to production developer tools. How to build OSS that gets adopted and sustained. My open source journey started in 2005 with Google Summer of Code, contributing to Mozilla and Linux kernel projects. Twenty years later, I've built production OSS tools, contributed to projects used by millions, and watched the open source ecosystem evolve from a fringe philosophy to the foundation of modern software development. Most open source projects fail. Not because the code is bad, but because the project decisions are wrong. Architecture that discourages contribution. Documentation that assumes too much context. Maintenance patterns that burn out creators. The technical quality of the code is necessary but nowhere near sufficient. This article covers the patterns I've seen work—and the anti-patterns I've learned to avoid—for building developer tools that get adopted and sustained. ## The First Month Determines Everything Open source projects have a critical window. In the first 30 days, potential contributors form lasting impressions about the project's quality, accessibility, and viability. Get these right early: ### A README That Sells and Teaches Your README is the landing page for your project. It needs to answer five questions in this order: 1. **What does this do?** (One sentence, no jargon) 2. **Why should I care?** (The problem it solves, with a concrete before/after) 3. **How do I try it?** (Copy-pasteable command that works in under 60 seconds) 4. **How does it work?** (Architecture overview for people who want to understand before committing) 5. **How do I contribute?** (Clear pointer to contribution guidelines) Most READMEs answer these in the wrong order—they start with architecture details or project history, burying the value proposition. When I built developer tools at Orangewood Labs, the RoboGPT SDK's README started with a single command: `pip install robogpt && robogpt demo`. A developer could see a robot arm execute a natural language command in under a minute. Everything else—architecture, configuration, advanced usage—was linked from the README, not crammed into it. ### The 10-Minute Contribution Test Clone your project on a fresh machine. Set a timer. If you can't go from `git clone` to a passing test suite in 10 minutes, your contribution barrier is too high. **What kills the 10-minute test**: - Undocumented system dependencies (specific library versions, system packages, database servers) - Configuration files that need manual editing before anything works - Build steps that require knowledge not in the README - Test suites that depend on external services or specific environment state **How to fix it**: - `Makefile` or equivalent with `make setup` and `make test` targets - Docker-based development environment as an alternative to local setup - CI configuration that doubles as setup documentation (if CI can build it from scratch, so can a contributor) - Seed data and test fixtures checked into the repository ### Contribution Guidelines That Enable CONTRIBUTING.md should cover: - **How to report bugs** (issue template with reproduction steps) - **How to propose features** (discussion format before implementation) - **How to submit code** (PR process, review expectations, merge criteria) - **Code style** (enforced by linters, not by reviewer opinions) - **Architecture map** (which files own which responsibilities—so contributors know where to make changes) The architecture map is the most underrated element. Contributors are willing to write code but not willing to spend hours understanding where their code should go. A simple diagram or list mapping capabilities to directories saves hours per contribution. ## Architecture for Contribution The way you structure code determines whether people can contribute to it. Open source projects need contribution-friendly architecture. ### Plugin Architecture The single most effective architectural pattern for open source developer tools is a plugin system. A stable core with extension points that contributors can build against without understanding the entire codebase. **Why it works**: - Contributors can add functionality without modifying core code - Plugin PRs have a smaller blast radius, making them easier to review - Users can compose exactly the features they need - The core team can focus on stability while the community drives feature breadth **Implementation pattern**: ``` core/ # Stable, slow-changing, tightly reviewed engine.rs # Core logic plugin.rs # Plugin interface definition plugins/ # Community-contributed, faster-moving plugin-a/ # Each plugin is self-contained plugin-b/ ``` The plugin interface is the critical design decision. Too narrow and plugins can't do anything interesting. Too broad and plugin changes break when the core changes. The sweet spot: define the interface in terms of capabilities (what plugins can do) rather than implementation details (how the core works). ### Modular Dependencies Every dependency you add is a dependency every contributor must understand. Minimize external dependencies in core code. When dependencies are necessary: - **Pin versions explicitly**. Don't use ranges. A contributor should get exactly the same build you get. - **Isolate dependency-heavy code** in separate modules. If your YAML parsing library has a security vulnerability, only one module needs updating. - **Avoid dependencies for trivial functionality**. If you need to left-pad a string, write the function. During my Google Summer of Code work on the Linux kernel, I learned the most extreme version of this principle: the kernel has essentially no external dependencies. Everything is built from source. While this is excessive for most projects, the principle—minimize and isolate dependencies—scales well. ### Clear Ownership Boundaries Every directory in your project should have a clear owner and purpose. When a contributor wants to add a feature, they should know immediately which directory to work in and who will review their PR. I use CODEOWNERS files religiously: ``` /core/ @maintainer-team /plugins/auth/ @auth-contributors /plugins/cache/ @cache-contributors /docs/ @docs-team ``` This isn't bureaucracy—it's routing. Contributors know who to ask questions and who will review their work. Reviewers know which changes require their attention. ## Community Building That Scales Code alone doesn't build community. Deliberate community architecture does. ### The Contributor Ladder Define clear progression paths for contributors: **Level 1: User** → Files issues, asks questions, gives feedback **Level 2: Contributor** → Submits PRs, reviews others' code, improves documentation **Level 3: Maintainer** → Merges PRs, triages issues, owns a module or plugin **Level 4: Core team** → Makes architectural decisions, manages releases, sets direction Each level should have explicit criteria for advancement. "Submit 5 reviewed PRs and maintain a plugin for 3 months" is better than "demonstrate sustained contribution" because it's objective and achievable. ### Good First Issues Label issues explicitly with `good-first-issue` and make them genuinely good first issues: - Self-contained (don't require understanding the whole codebase) - Well-specified (clear acceptance criteria, not "improve X") - Appropriately sized (4-8 hours of work, not 4 days) - Mentored (a maintainer is assigned to guide the contributor) At Mozilla, the `good-first-bug` system was how I entered open source in 2005. The bug I picked was small—a CSS rendering edge case—but it came with a mentor who walked me through the codebase, the testing approach, and the review process. That mentorship converted me from a user to a multi-year contributor. The pattern works because it invests in people, not just code. ### Responsive Maintenance The number one factor in open source project health isn't code quality—it's response time. A project where issues get triaged within 48 hours and PRs get reviewed within a week feels alive. A project where issues sit for months feels abandoned, regardless of code quality. **The maintenance SLA I target**: - Issue triage: 48 hours (acknowledge, label, assign—not necessarily fix) - PR review: 5 business days for initial feedback - Release cadence: Monthly for active projects, quarterly for stable ones - Security patches: 24 hours for critical, 72 hours for non-critical If you can't maintain these SLAs alone, recruit co-maintainers before the backlog becomes unmanageable. A burnt-out solo maintainer is the most common open source failure mode. ## Sustainability Patterns Open source sustainability is the problem nobody wants to talk about until it's too late. ### License Choice Matters For developer tools, I recommend: - **MIT/Apache 2.0**: Maximum adoption, minimum friction. Use this when adoption is more important than preventing proprietary use. - **MPL 2.0**: File-level copyleft. Changes to your files must be open, but your tool can be included in proprietary projects. Good middle ground. - **AGPL**: If the tool is a service (SaaS, API), this ensures modifications are shared. Use when preventing proprietary forks of services matters. License choice affects contribution patterns. Permissive licenses attract corporate contributors who need legal clarity. Copyleft licenses attract ideological contributors who value reciprocity. Neither is wrong—know your audience. ### The Dual-Track Model Sustaining open source long-term usually requires some form of economic model: **Open core**: Core tool is OSS, advanced features are commercial. Works well for developer tools where individual use is free but team/enterprise features have value. **Services**: The tool is fully open source. You sell hosting, support, or managed versions. Works when operational complexity is high. **Sponsorship**: Corporate sponsors fund development in exchange for influence over the roadmap. Works for infrastructure projects that large companies depend on. The model you choose should align with how your users derive value. If value comes from using the tool, charge for features. If value comes from not managing the tool, charge for management. If value comes from the ecosystem, seek sponsorship. ### Documentation as Infrastructure Documentation is not a nice-to-have for open source projects. It's infrastructure—as critical as CI pipelines and test suites. **The documentation stack**: - **README**: Quick start and project overview (under 500 words) - **Tutorial**: Guided walkthrough of a real use case (30-60 minutes) - **How-to guides**: Task-oriented instructions for specific goals - **Reference**: Complete API/configuration documentation (auto-generated where possible) - **Architecture**: Design decisions and system overview for contributors Each type serves a different audience at a different stage of their journey. Most projects have reference documentation but lack tutorials and how-to guides—the exact types that convert users into contributors. ## The Long Game The projects I've worked on that lasted—Mozilla, Linux kernel contributions, production tools at multiple companies—share common traits. They prioritize contribution over control. They invest in people, not just code. They accept that sustainable pace beats heroic effort. Open source is not a development methodology. It's a community architecture. Build the architecture deliberately, maintain it consistently, and the code will follow. --- ## How Nykaa Scaled to IPO: 500% Traffic Growth, Platform Migration, 99.99% Uptime URL: https://www.dipankar.name/writings/engineering-nykaa-zero-to-ipo/ Date: 2026-03-10 Tags: e-commerce, platform-architecture, startup-to-ipo, scalability, engineering-leadership, outcomes-and-impact, performance-optimization The engineering decisions behind Nykaa from startup to IPO — Magento to custom Python, in-memory cart service, Kong API gateway, 500% traffic growth, 99.99% uptime. Between 2017 and 2021, I served as Principal Engineering Consultant at Nykaa, India's largest beauty and personal care e-commerce platform. During that period, the platform went through 500% traffic growth, a complete platform migration, and ultimately an IPO that valued the company at $13 billion. This article documents the engineering decisions that enabled that trajectory—what we got right, what we got wrong, and what I'd do differently. This isn't a success narrative. It's a technical post-mortem of a scaling journey, written for engineering leaders facing similar growth challenges. ## The Starting State When I joined Nykaa in 2017, the platform was a monolithic PHP application running on a small cluster of EC2 instances. It worked, but it had the predictable problems of early-stage e-commerce platforms: - **Deployment frequency**: Once a week, usually on Sundays, with the entire engineering team on standby. - **Incident rate**: 2-3 production incidents per week, mostly related to database overload during flash sales. - **Feature velocity**: 4-6 weeks from concept to production for standard features. - **Scalability ceiling**: The platform could handle approximately 50K concurrent users before degrading. The business was growing at 80% year-over-year. The platform couldn't keep up. The question wasn't whether to re-architect—it was how to do it without slowing down a business that couldn't afford to slow down. ## Decision 1: Strangler Fig Migration **The decision**: Migrate from monolith to microservices incrementally using the strangler fig pattern, rather than attempting a complete rewrite. **Why this was right**: A full rewrite would have taken 12-18 months and frozen feature development. With 80% growth, that was commercially unacceptable. The strangler fig approach let us extract services one at a time while the monolith continued to serve traffic. **How we did it**: 1. **Identified extraction boundaries** based on business capability, not technical convenience. The first service we extracted wasn't the easiest—it was catalog search, because search latency directly impacted conversion. 2. **Built an API gateway** that routed traffic between the monolith and new services. This was the critical enabler—it let us redirect traffic at the URL level, making migrations invisible to the frontend. 3. **Established a service template** with standardized observability, deployment, and error handling. Every new service started from the same base, reducing per-service setup time from weeks to hours. **The outcome**: Over 18 months, we extracted 12 core services from the monolith. Feature velocity improved from 6 weeks to 2 weeks. Deployment frequency went from weekly to daily. **What I'd do differently**: We extracted services too granularly in some cases, creating distributed monolith problems. Three of our "microservices" were later consolidated because they were always deployed together and shared a data model. I'd be more aggressive about keeping related functionality in the same service. ## Decision 2: Performance as a Feature **The decision**: Treat page load time as a product feature with specific targets, dedicated engineering time, and business metric correlation. **The context**: In 2018, our mobile web experience had a 6-second time to interactive. India's e-commerce market is mobile-first—80%+ of Nykaa's traffic was mobile. Six seconds was losing customers before they saw a product. **The target**: Sub-2-second time to interactive on a mid-range Android device over 4G. **The approach**: - **Measured the business impact first**. We ran a controlled experiment: serve a percentage of users an intentionally faster (pre-cached) version of the homepage. The result—every 100ms of improvement correlated with a 0.8% increase in session duration and a 0.3% increase in add-to-cart rate. This turned performance optimization from a technical initiative into a revenue initiative. - **Attacked the critical rendering path**. Server-side rendering for above-the-fold content. Lazy loading for everything below. Image optimization pipeline that served WebP with appropriate dimensions based on device. - **Built a performance budget system**. Every new feature had a performance budget. If adding a recommendation widget increased page weight by 200KB, the team had to find 200KB of savings elsewhere or get explicit approval for the regression. - **Continuous monitoring**. Real user monitoring (RUM) dashboards displayed p50/p95/p99 load times segmented by device type, network, and geography. Any regression triggered an automated alert and a mandatory investigation. **The outcome**: TTI went from 6 seconds to 1.8 seconds over 6 months. During the same period, mobile conversion rate improved by 15%. The CFO became the performance team's biggest advocate—unusual for a cost center. **The lesson**: Performance work that's connected to revenue metrics gets funded indefinitely. Performance work pitched as "technical improvement" gets one quarter of investment and then loses priority. ## Decision 3: Flash Sale Architecture **The decision**: Build a dedicated flash sale system with pre-computed inventory, edge caching, and queue-based checkout rather than scaling the general e-commerce platform for peak load. **The context**: Nykaa's flash sales generated 10-20x normal traffic in 15-minute windows. Scaling the entire platform for these peaks was cost-prohibitive and architecturally wasteful—the sale traffic pattern was fundamentally different from browsing traffic. **The architecture**: - **Pre-computed product pages**: Flash sale items were rendered to static HTML and pushed to CDN edge nodes 30 minutes before the sale. The product page served during a flash sale was essentially a static file, not a database query. - **Inventory management via Redis**: Flash sale inventory was loaded into Redis with atomic decrement operations. No database involved in the hot path. When Redis inventory hit zero, the CDN served a "sold out" page. - **Queue-based checkout**: Instead of processing checkouts in real-time during the sale (which would overwhelm the payment system), users entered a checkout queue. The queue processed orders at a rate the downstream systems could handle, typically 500 orders per second. - **Graceful degradation**: If any component failed, the system defaulted to "sold out" rather than erroring. A false "sold out" is recoverable (restock and re-announce). A checkout error during a flash sale is a customer service nightmare. **The outcome**: We handled 500K concurrent users during a major sale event with zero downtime. The previous architecture had crashed at 50K. Cost per transaction during flash sales dropped 85% because we weren't over-provisioning the entire platform. **What I'd do differently**: The queue-based checkout introduced a UX challenge—users didn't know their position or estimated wait time. Adding a position indicator and time estimate would have reduced cart abandonment in the queue significantly. ## Decision 4: Observability-First Development **The decision**: Mandate that every new service ship with full observability before accepting any traffic. **The context**: By 2019, we had 12 microservices, and debugging cross-service issues was consuming 30% of senior engineering time. We had logging but not tracing. We had metrics but not correlation. Every incident required manual log-grepping across multiple services. **The standard**: Every service must ship with: 1. **Distributed tracing** using OpenTelemetry, with trace IDs propagated across all service calls 2. **Structured logging** in JSON format with standardized fields (request_id, user_id, service_name, latency_ms) 3. **RED metrics** (Rate, Errors, Duration) exposed via Prometheus endpoints 4. **Health checks** that test actual dependencies, not just process liveness 5. **Runbooks** for the top 3 expected failure modes **The enforcement mechanism**: Services that didn't meet the observability standard weren't allowed to register in the service mesh. No observability, no traffic. This was controversial—it slowed initial service deployment by 2-3 days. But it eliminated the "we can add monitoring later" pattern that had created our debugging crisis. **The outcome**: Mean time to detection (MTTD) for incidents dropped from 12 minutes to under 2 minutes. Mean time to resolution (MTTR) dropped from 90 minutes to 25 minutes. The 30% senior engineer time spent on debugging was redirected to feature work. ## Decision 5: Database Strategy for Scale **The decision**: Adopt a polyglot persistence strategy with clear guidelines for when to use each technology. **The monolith's approach**: Everything in MySQL. Product catalog, user sessions, order history, inventory, search indices—all in a single MySQL cluster. **The new strategy**: | Data Type | Technology | Why | |-----------|-----------|-----| | Product catalog | PostgreSQL | Complex queries, JSONB for flexible attributes | | User sessions | Redis | Ephemeral, high-frequency reads | | Search | Elasticsearch | Full-text, faceted search | | Order history | PostgreSQL (separate cluster) | Transactional integrity, audit requirements | | Inventory | Redis (primary) + PostgreSQL (source of truth) | Atomic operations at flash sale speed, durable record | | Analytics | ClickHouse | Columnar storage for high-volume event data | | Cache | Redis + CDN | Multi-layer caching for read-heavy traffic patterns | **The critical rule**: Every data store had a designated owner team and documented write patterns. No service could write to another service's data store. This prevented the distributed data coupling that kills microservice architectures. **The outcome**: Database-related incidents dropped from 2-3 per week to less than 1 per month. Individual data stores could be scaled, optimized, and maintained independently. ## The Numbers By the time of Nykaa's IPO filing in 2021, the engineering metrics told the story: | Metric | 2017 | 2021 | Change | |--------|------|------|--------| | Deploy frequency | Weekly | 15-20/day | ~100x | | Lead time (commit to prod) | 7 days | 45 minutes | ~220x | | Change failure rate | 18% | 2.1% | ~9x improvement | | MTTR | 90 min | 25 min | ~3.5x improvement | | Concurrent user capacity | 50K | 500K+ | 10x | | Mobile TTI | 6s | 1.8s | 3.3x improvement | | Platform uptime | 99.5% | 99.99% | — | | Feature lead time | 6 weeks | 2 weeks | 3x | These aren't aspirational numbers or benchmarks. They're the actual measurements from the monitoring systems we built. ## What I Got Wrong **Over-investment in microservice granularity**: Some services should have stayed together. The overhead of managing 12 services was manageable; the trajectory toward 30+ was not. **Under-investment in developer experience**: We built great production systems but neglected the local development environment. Engineers spent too long setting up dependencies and too little time coding. A docker-compose-based local environment should have been a top-3 priority from day one. **Late adoption of feature flags**: We should have implemented feature flagging in the first month, not the first year. The ability to decouple deployment from release would have saved us several painful rollbacks. **Insufficient documentation of trade-offs**: We documented decisions but not the alternatives we rejected. When new engineers asked "why didn't we use X?" we often couldn't articulate the reasons, leading to re-litigation of settled decisions. ## The Transferable Lessons These insights apply beyond e-commerce: 1. **Connect engineering metrics to business metrics early**. The moment engineering work has a revenue number attached, organizational support follows. 2. **Migrate incrementally, not heroically**. The strangler fig pattern is slower but orders of magnitude less risky than a rewrite. 3. **Build for the peak, serve the baseline efficiently**. Flash sale architecture is a specific example of a general principle: design for your worst case separately from your normal case. 4. **Observability is not optional**. It's a prerequisite for production traffic. Enforce this mechanically, not culturally. 5. **Boring technology choices, exceptional execution**. None of our technology choices were cutting-edge. PostgreSQL, Redis, Elasticsearch—all proven, well-understood tools. The value was in how we applied them, not what we chose. An IPO is a business milestone, not an engineering one. But the engineering decisions made years earlier determined whether the platform could support the business growth that made the IPO possible. Scaling isn't a single decision—it's a sequence of deliberate choices, each building on the last, each accepting specific trade-offs for specific gains. --- ## Measuring Engineering Impact: Beyond Lines of Code and Story Points URL: https://www.dipankar.name/writings/measuring-engineering-impact/ Date: 2026-02-15 Tags: engineering-metrics, engineering-leadership, productivity, outcomes-and-impact, team-performance, engineering-management Dipankar Sarkar presents a framework for measuring engineering impact that connects technical work to business outcomes. Move beyond vanity metrics to measure what actually matters. Every engineering organization I've joined had metrics. Velocity charts. Sprint burndown. Lines of code. Commit frequency. And in every case, these metrics measured activity while saying nothing about impact. At Nykaa, we had a team with the highest velocity in the organization. They completed more story points per sprint than any other team. They were also responsible for the most production incidents, the highest bug count, and the slowest time-to-resolution. High activity, negative impact. The problem isn't measurement—it's measuring the wrong things. Engineering organizations need metrics that connect technical work to business outcomes. Here's the framework I've developed across 18+ years of building production systems. ## The Metrics Hierarchy Engineering metrics exist at four levels. Most organizations measure only the bottom two and wonder why their metrics don't correlate with business results. ### Level 1: Activity Metrics (Least Valuable) **What they measure**: How much work is happening. - Lines of code written - Commits per day - Story points completed - PRs merged - Tickets closed **Why they fail**: Activity metrics are trivially gameable and inversely correlated with the work that matters most. The engineer who spends three days thinking through an architecture decision before writing 50 lines of elegant code delivers more value than the one who writes 500 lines of code that needs to be rewritten next quarter. **When they're useful**: Never as primary metrics. Occasionally as anomaly detectors—a sudden drop in PR activity from a usually productive engineer might indicate a blocker, burnout, or unclear requirements. ### Level 2: Process Metrics (Necessary but Insufficient) **What they measure**: How well the engineering process functions. - **Lead time**: Time from commit to production deployment - **Deploy frequency**: How often you ship to production - **Change failure rate**: Percentage of deployments causing incidents - **Mean time to recovery (MTTR)**: How quickly you restore service after an incident These are the DORA metrics, and they're valuable. They measure engineering capability—the ability to ship changes quickly and safely. But they don't measure whether those changes mattered. A team with excellent DORA metrics that ships features nobody uses is still failing. Process metrics tell you that you can deliver. They don't tell you that you're delivering the right things. ### Level 3: Output Metrics (Getting Warmer) **What they measure**: What engineering produced. - Features shipped to users - API endpoints with active consumers - System reliability (uptime, error rates, latency percentiles) - Technical debt reduced (measured by concrete indicators, not gut feel) - Platform capabilities enabled (e.g., "we can now A/B test any checkout flow") Output metrics are where most sophisticated engineering organizations stop. They're meaningful—they represent real work delivered to real users. But they still have a blind spot: they don't measure whether the output changed anything. ### Level 4: Outcome Metrics (Most Valuable) **What they measure**: What changed because of engineering's work. - Revenue impact of shipped features - User engagement changes attributable to technical improvements - Cost reduction from infrastructure optimization - Time-to-market improvement for product teams - Customer satisfaction changes linked to reliability improvements **Why most organizations don't measure these**: Outcome metrics require collaboration between engineering, product, and data teams. They require attribution—connecting a business result to a specific technical change. This is hard, imperfect, and sometimes contentious. But imperfect outcome measurement is infinitely more valuable than precise activity measurement. A rough estimate that "the checkout optimization shipped by the platform team increased conversion by 0.3%" is more useful than knowing the team completed 47 story points. ## Implementing the Framework ### Step 1: Define Your Outcome Map Every engineering team exists to drive specific business outcomes. Make these explicit: **Platform team**: System reliability → user trust → retention → revenue **Feature team**: Feature delivery → user engagement → activation → revenue **Infrastructure team**: Developer productivity → faster shipping → more experiments → better product-market fit **Data team**: Analytics capabilities → better decisions → optimized operations → margin improvement At Nykaa, the platform team's outcome map was concrete: every 10ms of homepage latency reduction correlated with a 0.1% increase in conversion rate. This made infrastructure optimization directly measurable in revenue terms. When I proposed a CDN migration that would cost $30K/month but reduce p95 latency by 80ms, the business case was trivial: the estimated conversion improvement was worth 20x the CDN cost. ### Step 2: Instrument the Connections Outcome measurement requires instrumentation at every level of the stack: **Technical instrumentation**: Standard observability—latency, error rates, throughput. This is table stakes. **Product instrumentation**: Feature flags with analytics. Every feature ships with tracking that measures adoption, engagement, and the product metric it's targeting. **Business instrumentation**: Revenue attribution, cost tracking, and conversion funnels that connect to technical changes. The critical integration: deployment events correlated with business metrics. When you can overlay "deployed checkout optimization v2" on your conversion rate graph, you can see impact directly. Tools like feature flags with analytics built in make this straightforward. ### Step 3: Establish Attribution Conventions Perfect attribution is impossible. Multiple changes ship simultaneously. External factors affect metrics. User behavior is noisy. Establish conventions that are good enough: **Direct attribution**: The feature was behind a feature flag. The flag was enabled. The metric changed. Confidence is high. **Correlated attribution**: The change shipped and the metric moved in the expected direction within the expected timeframe. Confidence is medium. **Inferred attribution**: The change was part of a bundle of improvements. The combined impact is measurable but individual contributions are estimated. Confidence is low but still useful for prioritization. At Hike, the ML team's recommendation system improvements were measured through direct attribution—A/B tests with clear control groups. But the infrastructure team's latency improvements were correlated attribution at best. We accepted this asymmetry and used it for directional guidance rather than precise accounting. ### Step 4: Build the Reporting Cadence **Weekly**: Process metrics (DORA). These move fast and indicate operational health. **Monthly**: Output metrics. Features shipped, reliability maintained, technical debt addressed. This is the engineering team's primary reporting cadence. **Quarterly**: Outcome metrics. Business impact of engineering work over the quarter. This connects engineering effort to organizational goals and informs the next quarter's prioritization. **Annually**: Strategic review. Which engineering investments produced the highest outcome-to-effort ratio? Which areas deserve more investment? Which should be wound down? ## The Metrics That Actually Changed My Teams Beyond the framework, here are specific metrics that drove the most behavioral improvement in teams I've led: ### Time to First Meaningful Contribution **What it measures**: How long it takes a new engineer to merge their first PR that affects production behavior (not a docs fix or config change). **Why it matters**: This metric is a proxy for onboarding effectiveness, codebase quality, and team collaboration culture. A team where new engineers ship in their first week has good documentation, approachable code, and supportive teammates. **Target**: Under 5 business days for experienced engineers, under 10 for early-career. At Orangewood Labs, this metric exposed that our robotics SDK had an undocumented dependency on a specific ROS version that took every new engineer 3 days to discover and resolve. Fixing the documentation cut time-to-first-contribution from 12 days to 4. ### Escaped Defect Rate **What it measures**: Bugs found in production versus bugs caught before deployment. **Why it matters**: It measures the effectiveness of your entire quality pipeline—code review, testing, staging environments, and monitoring. A decreasing escaped defect rate means your prevention systems are improving. **Target**: Below 10% for mature teams (90%+ of bugs caught before production). ### Decision-to-Deploy Latency **What it measures**: Time from "we've decided to build X" to "X is in production." **Why it matters**: This captures everything—planning overhead, development time, review bottlenecks, deployment pipeline speed, and organizational friction. Unlike pure lead time (which measures only the technical pipeline), this includes the human and organizational delays. **Target**: Under 2 weeks for standard features in a mature organization. ### Recovery Learning Rate **What it measures**: For each incident category, how much faster was the team's response the second time it occurred? **Why it matters**: Every team has incidents. The best teams learn from them measurably. If your median response time for database issues was 45 minutes last quarter and 20 minutes this quarter, your learning systems are working. ## Common Pitfalls ### Goodhart's Law "When a measure becomes a target, it ceases to be a good measure." This applies to every metric in this framework. **Mitigation**: Use a balanced scorecard approach. No single metric is a target. The portfolio of metrics across all four levels provides a holistic picture that's resistant to gaming. ### Survivor Bias in Metrics You measure the features you ship but not the features you don't. The most impactful engineering decision might be saying no to a feature that would have been expensive to maintain. **Mitigation**: Track "complexity avoided" as an explicit metric. When you simplify a design, eliminate a dependency, or reject a feature for maintenance reasons, document the estimated ongoing cost you avoided. ### The McNamara Fallacy Measuring what's easy to measure, ignoring what's hard, and then assuming what's hard to measure isn't important. **Mitigation**: Accept imprecise measurement of important things over precise measurement of unimportant things. A rough revenue attribution is better than an exact commit count. ## Making It Real Start with one outcome metric per team. Don't try to implement the entire framework at once. Pick the metric that most directly connects engineering work to a business result, instrument it, and report on it for one quarter. When I introduced outcome metrics at Nykaa, we started with a single metric for the platform team: p95 API latency correlated with checkout conversion. This one metric changed how the team prioritized work more than any process improvement I'd implemented before. When engineers could see that their optimization work translated directly to revenue, intrinsic motivation replaced the need for velocity tracking. That's the real power of measuring impact: when engineers understand how their work matters, you spend less time managing and more time enabling. --- ## LLM Provider Risk Management: Multi-Provider Strategy for Banks and Regulated Industries URL: https://www.dipankar.name/writings/llm-provider-risk-management/ Date: 2026-01-27 Tags: llm-provider-risk, third-party-risk, ai-governance, financial-services, data-privacy, vendor-management, regulatory-compliance How to manage LLM provider risk in regulated industries. Trust Boundaries Model, multi-provider strategy, fallback patterns, and compliance mapping for financial services AI. LLM providers create a unique risk profile for regulated firms. They are simultaneously an outsourcing dependency (SS2/21 applies), a resilience risk (PS21/3 applies), and a data protection concern (UK GDPR applies). A single provider relationship triggers three regulatory frameworks. More challenging: LLM providers can change your AI agent's behaviour without your involvement. A model update, a policy change, a training data refresh—any of these can alter how your agent responds to customers. You deploy on Tuesday; the provider updates on Wednesday; Thursday your agent behaves differently. This guide presents frameworks for managing LLM provider risk: the **Trust Boundaries Model** for data protection, **Reversible Tokenisation** for privacy, and **Multi-Provider Strategy** for resilience. ## The Dual Risk Problem LLM providers represent two risks simultaneously: ### Outsourcing Risk Under SS2/21, critical service providers must be: - Subject to due diligence - Covered by appropriate contracts - Monitored for performance - Subject to exit planning LLM providers fit this framework. But they're not traditional outsourcers: - You can't audit their models - You can't control their update schedule - You can't prevent unilateral changes - Your exit options are limited ### Resilience Risk Under PS21/3, Important Business Services must meet impact tolerances. If your AI agent is an IBS—or supports one—LLM provider availability is a resilience concern. But LLM providers are: - Concentrated (few major providers) - Interconnected (shared infrastructure) - Non-substitutable quickly (prompts are provider-specific) Traditional resilience planning assumes you can fail over. LLM failover is harder than it looks. ## The Trust Boundaries Model Not all data is equal. Not all destinations are equal. The Trust Boundaries Model classifies both: ### Data Trust Levels **Untrusted Input** Customer input, external data. May contain: - Prompt injection attempts - Malformed data - PII that shouldn't be shared - Malicious content Treatment: Validate, sanitise, classify before any processing. **Semi-Trusted Input** LLM responses, third-party data. May contain: - Hallucinations - Inappropriate content - Outdated information - Policy violations Treatment: Filter, validate, constrain before use. **Trusted Data** Internal systems, policy engine. Should be: - Verified at source - Logged for audit - Protected in transit Treatment: Use with appropriate logging. ### Destination Trust Levels **External (LLM Provider)** Data sent to external providers is: - Subject to provider policies - Potentially retained for training - Transmitted internationally - Outside your direct control Minimise what you send. Assume retention unless contractually excluded. **Internal** Data within your infrastructure is: - Subject to your policies - Under your control - Within your jurisdiction Log appropriately. Apply internal controls. ### Trust Boundary Enforcement ``` Customer Input → [Validate] → [Sanitise] → [Classify] ↓ ┌─────────────┐ │ PII │ │ Detection │ └─────────────┘ ↓ ┌─────────────┐ │ Tokenise │ │ PII │ └─────────────┘ ↓ ════════ TRUST BOUNDARY ════════ ↓ ┌─────────────┐ │ LLM │ │ Provider │ └─────────────┘ ↓ ════════ TRUST BOUNDARY ════════ ↓ ┌─────────────┐ │ Filter │ │ Response │ └─────────────┘ ↓ ┌─────────────┐ │ De-tokenise │ │ PII │ └─────────────┘ ↓ Customer Response ``` The trust boundary is explicit. Data is transformed crossing it. ## Reversible Tokenisation The best way to protect PII sent to LLM providers is not to send it. Reversible tokenisation replaces PII with tokens before LLM calls, then restores it after. ### What to Tokenise | Data Type | Treatment | Rationale | |-----------|-----------|-----------| | Account numbers | Tokenise → [ACCT_1] | Never needed in LLM response | | Sort codes | Tokenise → [SORT_1] | Never needed in LLM response | | Names | Context-dependent | Sometimes needed for personalisation | | Addresses | Mask | Rarely needed in full | | Financial amounts | Pass through | Often needed for meaningful response | | Phone numbers | Tokenise | Never needed in response | | Email addresses | Tokenise | Never needed in response | | Dates | Context-dependent | Sometimes needed | ### Token Map Architecture ```python class TokenMap: def __init__(self, session_id: str): self.session_id = session_id self.mappings = {} # token -> original value self.created_at = datetime.now() def tokenise(self, value: str, category: str) -> str: token = f"[{category}_{len(self.mappings) + 1}]" self.mappings[token] = value return token def detokenise(self, text: str) -> str: for token, value in self.mappings.items(): text = text.replace(token, value) return text def purge(self): """Called at session end - removes all mappings""" self.mappings.clear() ``` **Critical properties**: - Token maps never leave your infrastructure - Maps are session-scoped and purged at session end - Tokens are meaningless without the map - LLM sees only tokens, never real values ### Example Flow **Customer says**: "Transfer £500 from my account 12345678 to John Smith at 87654321" **Tokenised (sent to LLM)**: "Transfer £500 from my account [ACCT_1] to [NAME_1] at [ACCT_2]" **LLM responds**: "I'll transfer £500 from [ACCT_1] to [NAME_1]'s account [ACCT_2]. Please confirm." **Detokenised (shown to customer)**: "I'll transfer £500 from 12345678 to John Smith's account 87654321. Please confirm." The LLM never saw the real account numbers. Your customer sees a natural response. ## Multi-Provider Strategy Relying on a single LLM provider creates concentration risk. Multi-provider strategy provides resilience. ### Architecture ``` ┌─────────────────────────────────────────────────────┐ │ LLM Gateway │ │ ┌─────────────────────────────────────────────┐ │ │ │ Provider Router │ │ │ │ - Health monitoring │ │ │ │ - Load balancing │ │ │ │ - Failover logic │ │ │ │ - Cost optimisation │ │ │ └─────────────────────────────────────────────┘ │ │ │ │ │ │ │ ┌──────┴────┐ ┌──────┴────┐ ┌──────┴────┐ │ │ │ Primary │ │ Secondary │ │ Fallback │ │ │ │ Provider │ │ Provider │ │ (Local) │ │ │ │ (OpenAI) │ │ (Anthropic)│ │ (Ollama) │ │ │ └───────────┘ └───────────┘ └───────────┘ │ └─────────────────────────────────────────────────────┘ ``` ### Provider-Agnostic Prompts Prompts must work across providers. This means: - Avoid provider-specific features - Test prompts on all providers - Accept some capability reduction for portability - Version prompts by provider if necessary ### Health Monitoring Monitor each provider continuously: ``` Health check every 30 seconds: - Latency (P50, P95, P99) - Error rate - Cost per request - Rate limit headroom ``` Unhealthy provider → route to alternative → alert operations. ### Failover Logic ```python def route_request(request: LLMRequest) -> LLMResponse: for provider in get_healthy_providers(): try: response = provider.call(request) validate_response(response) return response except ProviderError: mark_unhealthy(provider) continue # All providers failed return graceful_degradation(request) ``` ### Quality Validation Different providers give different responses. Validate quality: - Response completeness - Factual accuracy (where verifiable) - Tone and appropriateness - Policy compliance Accept quality variation within bounds. Reject responses that fail validation. ## Provider Change Management LLM providers update models without your approval. Manage this: ### Change Detection Monitor for changes: - Model version (if exposed) - Response patterns - Latency characteristics - Error rates When patterns shift, investigate before assuming your code is wrong. ### Regression Testing Maintain a test suite that runs: - After deployments (your changes) - Daily (detect provider changes) - On alert (investigate issues) Compare results against baseline. Flag significant deviations. ### Rollback Capability If a provider change harms your service: - Fail over to alternative provider - Or: use cached model version (if available) - Or: graceful degradation Have a plan before you need it. ## International Data Transfers Most LLM providers are US-based. Sending data to them is an international transfer under UK GDPR. ### Transfer Mechanisms **Standard Contractual Clauses (SCCs)** Most providers offer SCCs. Verify they're current and appropriate. **Transfer Impact Assessments (TIAs)** Document the risk of transfer and mitigations applied: - What data is transferred? - What protections does the provider offer? - What's the legal access risk in the destination? - What mitigations have you applied? ### Architectural Mitigations Reduce transfer risk through architecture: **Tokenisation**: PII never leaves your jurisdiction **Zero-retention agreements**: Provider deletes immediately after processing **Regional endpoints**: Use EU/UK endpoints where available **Encryption**: Encrypt in transit and verify provider practices ### Documentation Maintain records for regulatory enquiry: - Data flows mapped - Legal basis documented - SCCs in place - TIA completed - Mitigations implemented ## Exit Planning SS2/21 requires exit plans for critical providers. For LLM providers: ### Exit Triggers Define what triggers exit consideration: - Regulatory direction - Security breach - Unacceptable cost increase - Quality degradation - Provider instability ### Exit Timeline Be realistic: - Immediate (days): Fail over to alternative provider, accept degraded service - Short-term (weeks): Adapt prompts for alternative provider, validate quality - Medium-term (months): Full testing, gradual migration, monitoring ### Concentration Risk Avoid single-provider dependency: - Tier 1 agents: Multi-provider mandatory - Tier 2 agents: Multi-provider recommended - Tier 3 agents: Single provider acceptable with monitoring Document concentration and review quarterly. ## Common Provider Risk Failures ### Failure 1: Treating Providers as Utilities "They're too big to fail." They're not. They have outages. They change policies. They sunset models. Plan for failure. ### Failure 2: Sending Everything Sending full customer context when a summary would do. Sending PII when tokens would work. Minimise by default. ### Failure 3: No Quality Monitoring Assuming provider quality is constant. It isn't. Monitor and detect degradation early. ### Failure 4: Lock-In Acceptance Provider-specific features that preclude alternatives. Acceptable trade-offs exist, but make them consciously. ### Failure 5: Exit Plans as Fiction Plans that exist on paper but haven't been tested. If you haven't failed over in a drill, you can't fail over in a crisis. ## When to Seek Expert Help LLM provider risk is complex and evolving. External expertise helps when: - **Establishing provider governance**: Get the framework right from the start - **Conducting due diligence**: Know what questions to ask - **Designing multi-provider architecture**: Resilience without excessive complexity - **Preparing for regulatory review**: Documentation that satisfies supervisors I help regulated firms manage LLM provider risk with frameworks that satisfy regulators while enabling innovation. [Get in touch →](/contact/) ## Related Reading - [AI Agent Governance for Financial Services](/writings/ai-governance-financial-services/) - Overall governance framework - [Defence in Depth for AI Agents](/writings/defence-in-depth-ai-agents/) - Control layers and kill switches - [AI Agent Safety: The Substrate Pattern](/writings/ai-agent-substrate-pattern/) - Execution envelope architecture --- *Dipankar Sarkar is a technology advisor specializing in AI risk management for regulated industries. He helps banks and insurers navigate LLM provider relationships with frameworks that satisfy regulators while enabling AI innovation. [Learn more →](/about/)* --- ## Remote-First Engineering Teams: Building Distributed Organizations That Deliver URL: https://www.dipankar.name/writings/remote-first-engineering-teams/ Date: 2025-12-20 Tags: remote-engineering, team-building, engineering-leadership, distributed-teams, async-communication, engineering-culture Dipankar Sarkar explains how to build remote-first engineering teams that outperform co-located ones. Communication architecture, async workflows, and trust systems for distributed teams. Most "remote" engineering teams are actually co-located teams with a remote option. The meeting culture, decision-making patterns, and communication norms are designed for people in the same room, with remote participants as second-class citizens. This creates a predictable failure mode: remote engineers feel disconnected, information flows through hallway conversations they're not part of, and leadership eventually concludes that "remote doesn't work." Remote doesn't work when you bolt it onto a co-located culture. Remote-first—where distributed is the default, not the exception—works extremely well when you design for it deliberately. I've built and led distributed teams across India, the US, and Japan. The teams that worked best weren't the ones with the best video conferencing setup. They were the ones with the best communication architecture. ## The Communication Architecture In co-located teams, communication architecture is implicit. Information flows through proximity: overhearing conversations, whiteboard sessions, lunch discussions. Remote-first teams need to make this architecture explicit. ### Principle 1: Write Everything Down The single most impactful change when going remote-first is shifting from verbal to written communication as the primary medium. Not as documentation—as the actual communication channel. **What this means in practice**: - **Design decisions** are written proposals, not meeting outcomes. An RFC (Request for Comments) document is circulated asynchronously. Comments are written. The decision is recorded in the document, not in someone's meeting notes. - **Status updates** are written, not spoken. A weekly written update from each team (3-5 bullet points: what shipped, what's blocked, what's next) replaces standup meetings for cross-team visibility. - **Context** is captured at the point of creation. When you make a decision, you write down why in the PR description, the ticket, or the design doc. Not later, not in a wiki—at the moment of the decision. At Orangewood Labs, we had engineers in Bangalore and partners in the US. The 10.5-hour time zone gap meant synchronous communication was limited to a 3-hour window. Writing everything down wasn't a nice-to-have—it was operational necessity. Design proposals were Google Docs with comment threads. Every technical decision had a written rationale. New engineers could reconstruct six months of context from the document trail alone. ### Principle 2: Async by Default, Sync by Exception Synchronous communication (meetings, calls, screen-shares) is expensive in remote teams. It requires schedule coordination across time zones, creates single points of failure (miss the meeting, miss the context), and produces information that's hard to reference later. **The async-first rule**: Any communication that doesn't require real-time back-and-forth should be asynchronous. This includes: - Code reviews (written comments, not live walkthroughs) - Design feedback (document comments, not design review meetings) - Status updates (written posts, not standup meetings) - Questions (posted in channels, not DMs or calls) **When sync is appropriate**: - Brainstorming sessions where rapid idea generation matters - Conflict resolution where tone and nuance are critical - Onboarding conversations where relationship building is the goal - Incident response where speed of coordination is paramount The ratio should be roughly 80% async, 20% sync. Most teams I've seen operate at 30/70 or worse, which is why remote feels exhausting—they're doing co-located communication patterns over video calls. ### Principle 3: Structured Information Channels In co-located offices, information finds its audience through proximity. Remotely, you need to design channels deliberately: **Broadcast channels** (one-to-many, low frequency): - Weekly team updates - Architecture decision records - Incident post-mortems **Discussion channels** (many-to-many, medium frequency): - RFC comment threads - Technical Q&A spaces - Project-specific channels **Direct channels** (one-to-one, as needed): - Mentoring conversations - Performance feedback - Sensitive topics **The critical rule**: Information that affects multiple people must go through broadcast or discussion channels, never direct channels. The moment important context lives only in a DM thread between two people, you've recreated the hallway conversation problem that remote-first was supposed to solve. ## Decision-Making in Distributed Teams Co-located teams can make decisions through conversation and consensus in a meeting room. Distributed teams need a more structured approach. ### The RFC Process Every significant technical decision goes through a written RFC: 1. **Author writes the proposal** (1-3 pages): Problem statement, proposed solution, alternatives considered, trade-offs accepted. 2. **Review period** (3-5 business days): Team members comment asynchronously. The author responds to comments and updates the proposal. 3. **Decision** (explicit, recorded): The decision-maker (tech lead, architect, or team) records the decision and rationale in the document. **Why this works remotely**: Everyone has equal access to the proposal regardless of time zone. Introverts who won't speak up in meetings can write thoughtful comments. The decision trail is permanent and searchable. At Hike, we used RFCs for all cross-team technical decisions. When I joined, decisions were made in meetings that the ML team often missed because they were in a different building. Switching to written RFCs didn't just improve remote collaboration—it improved all collaboration by making the decision process transparent and inclusive. ### Decision Escalation Not every decision needs an RFC. The framework: - **Reversible, low-impact**: Engineer decides, informs team in async update. - **Reversible, high-impact**: Engineer proposes in team channel, 24-hour comment period, proceeds if no objections. - **Irreversible, low-impact**: Tech lead approves, documented in ticket. - **Irreversible, high-impact**: Full RFC process. The key insight: categorize decisions by reversibility, not importance. Reversible decisions should be made quickly. Irreversible decisions deserve deliberation regardless of how "small" they seem. ## Trust Systems Remote teams run on trust. Without the visual cues of seeing someone at their desk, organizations often substitute surveillance—activity monitoring, mandatory camera-on meetings, frequent check-ins. This destroys the trust it claims to measure. ### Output-Based Accountability The only metric that matters for remote engineers: do they ship? **What I track**: - Commits merged to main (not commit count—meaningful contributions) - Code review turnaround time (responsiveness to teammates) - Design document contributions (thinking, not just coding) - Incident response participation (reliability as a teammate) **What I explicitly don't track**: - Hours online - Messages sent - Meeting attendance (beyond critical ones) - Response time to non-urgent messages This isn't hands-off management. I have weekly 1:1s with every direct report—30 minutes, focused on blockers, career growth, and feedback. But between those touchpoints, I trust people to manage their time. ### The Trust Ramp New hires in remote teams need a deliberate trust-building process: **Week 1-2**: High-touch onboarding. Daily sync calls. Pair programming sessions. The goal is relationship building, not productivity. **Week 3-4**: Structured tasks with clear success criteria. The engineer works independently but reports progress daily. The goal is demonstrating capability. **Month 2-3**: Increasing autonomy. Weekly check-ins replace daily ones. The engineer owns small features end-to-end. The goal is establishing working patterns. **Month 3+**: Full autonomy within the team's operating model. The engineer is trusted to manage their time, raise blockers proactively, and deliver on commitments. Rushing this ramp is the most common mistake I see in remote teams. A co-located engineer absorbs culture through osmosis. A remote engineer needs it delivered deliberately. ## Time Zone Strategy With teams spanning multiple time zones, you need an explicit strategy: ### The Overlap Window Identify the largest overlap window between your time zones. Protect it ruthlessly for synchronous collaboration: - All meetings happen in this window - It's the designated time for real-time pair programming - Incident escalation during this window gets voice calls, not text Outside the overlap window, everything is async. No exceptions. ### Follow-the-Sun Handoffs For teams with minimal overlap, design workflows that hand off across time zones: - Engineer A in IST works on a feature, leaves detailed notes at end of day - Engineer B in PST picks up from those notes, continues work, leaves their own notes - Each handoff is a written summary: what was done, what's next, what's blocked This pattern turns time zones from a liability into an asset—your team can make progress across 16+ hours of the day. ## Tools Are Secondary Every article about remote work leads with tools. I'm mentioning them last because they matter least. The best tools can't save a team with bad communication architecture, and adequate tools work fine with good processes. That said, the non-negotiable tools: - **Async writing**: Something with commenting and threading (Google Docs, Notion, or even GitHub Issues) - **Real-time chat**: For quick questions and social connection (Slack, Discord) - **Video calls**: For the 20% of communication that should be synchronous - **Shared codebase**: With good PR tooling (GitHub, GitLab) The specific tools matter far less than how you use them. A team with great async practices on mediocre tools will outperform a team with poor practices on best-in-class tools every time. ## When Remote-First Fails Remote-first isn't right for every situation: - **Early-stage exploration**: When you're still figuring out what to build, the bandwidth of co-located collaboration is hard to replace. I'd co-locate for the first 2-3 months of a new product. - **Hardware-dependent work**: Robotics at Orangewood required physical presence for testing. We made the software layer remote-first but kept the hardware team co-located. - **Cultural transformation**: If you're changing an organization's values or practices, the trust required is harder to build remotely. The honest assessment: remote-first is better for execution than exploration, better for steady-state than transformation, and better for experienced engineers than early-career ones. Design your organizational model accordingly. ## The Compound Effect The teams I've built remotely shipped as well or better than co-located teams. Not because remote is inherently superior, but because the discipline required to make remote work—written communication, explicit decisions, output-based trust—improves engineering organizations regardless of where people sit. Co-located teams can survive on implicit communication. Remote-first teams can't. That constraint, embraced rather than fought, produces clearer thinking, better documentation, and more inclusive decision-making. Build the communication architecture first. The rest follows. --- ## Defence in Depth for AI Agents: Kill Switches, Circuit Breakers, and Control Layers URL: https://www.dipankar.name/writings/defence-in-depth-ai-agents/ Date: 2025-12-09 Tags: ai-safety, kill-switch, circuit-breaker, defence-in-depth, ai-controls, financial-services, production-systems 5-layer kill switch architecture for production AI agents: application logic, tool execution, action policy, circuit breaker, and manual kill switch. Stop any agent in under 30 seconds. AI agents in production can fail in ways that affect thousands of customers simultaneously. A bug in traditional software affects users one at a time as they encounter it. An AI agent making wrong decisions processes its entire queue before anyone notices. The blast radius is different. The controls must be too. Defence in depth means no single control failure causes customer harm. Five layers of controls, independent kill switches, automatic circuit breakers, and graceful degradation work together to ensure that when—not if—something goes wrong, the impact is contained. ## The Five-Layer Control Framework Controls at a single point fail. Controls at multiple points provide defence in depth. ### Layer 1: Input Controls Control what enters the system. **Validation**: Reject malformed, out-of-range, or unexpected inputs before processing. ``` Input validation rules: - Schema validation (required fields, types) - Range validation (amounts within limits) - Format validation (dates, account numbers) - Relationship validation (from ≠ to account) ``` **Sanitisation**: Clean inputs that might manipulate LLM behaviour. - Strip control characters - Detect prompt injection patterns - Encode special characters - Truncate excessive length **Rate Limiting**: Prevent abuse and contain blast radius. - Per-customer limits (requests per minute) - Per-agent limits (total throughput) - Cost limits (LLM spend per hour) **Authentication**: Verify identity before processing. - Customer authentication - System-to-system authentication - Token validation and expiry ### Layer 2: Processing Controls Control what happens during execution. **Policy Enforcement**: Explicit rules the AI cannot override. ``` Policy rules (examples): - Maximum transaction amount: £10,000 - Restricted countries: [list] - Required fields for high-value: [list] - Prohibited actions: [list] ``` Policies are not prompts. They are code-enforced constraints that apply regardless of what the LLM outputs. **Context Boundaries**: Limit what the AI can access. - Customer can only access own data - Agent can only access required systems - PII is tokenised before LLM calls - Historical context is bounded **Transaction Limits**: Contain financial impact. - Single transaction limits - Daily aggregate limits - Automatic escalation above thresholds - Cooling-off periods for large decisions ### Layer 3: Output Controls Control what leaves the system. **Content Filtering**: Block harmful outputs. - Toxicity detection - Sensitive information scanning - Competitor/inappropriate content - Regulatory trigger phrases **Hallucination Detection**: Identify and handle confabulation. - Response grounding verification - Confidence scoring - Source citation requirements - Unknown acknowledgment patterns **Compliance Screening**: Ensure outputs meet requirements. - Regulatory disclosure requirements - Fair treatment language - Mandatory warnings - Prohibited claims ### Layer 4: Decision Controls Control significant decisions. **Human-in-the-Loop**: Require human approval for high-impact decisions. ``` Escalation triggers: - Amount > threshold - Customer flagged vulnerable - Low confidence score - First-time action type - Regulatory sensitive area ``` **Approval Workflows**: Structure human review. - Clear presentation of AI recommendation - All relevant context visible - Explicit approve/reject/modify options - Decision recorded with rationale **Override Capability**: Allow human correction. - Clear override interface - Override recorded and audited - Learning from overrides ### Layer 5: System Controls Control the system itself. **Kill Switches**: Stop AI agents rapidly. (Detailed below) **Circuit Breakers**: Automatic protection when metrics degrade. (Detailed below) **Monitoring and Alerting**: Detect problems early. - Real-time dashboards - Threshold-based alerts - Anomaly detection - On-call escalation **Audit Logging**: Record everything for forensics. - Every decision logged - Every control execution logged - Tamper-evident storage - Retention per regulatory requirements ## Kill Switch Architecture A kill switch must stop an AI agent rapidly and reliably. This requires: ### Non-Negotiable Requirements 1. **Activation < 60 seconds**: From decision to effect 2. **Independence**: Kill switch infrastructure separate from AI system 3. **Multiple Authorisers**: Not dependent on single person 4. **Always Available**: Works even when other systems are degraded 5. **Audit Trail**: Every activation logged with reason 6. **Regular Testing**: Quarterly drills minimum ### Five-Level Kill Switch Hierarchy Different situations require different scope: **Level 1: Global** Disables all AI agents across the platform. Nuclear option for systemic issues. **Level 2: Agent-Specific** Disables a single AI agent type. Use when one agent is misbehaving but others are fine. **Level 3: Feature-Specific** Disables a specific capability within an agent. Use for targeted issues. **Level 4: Customer-Specific** Disables AI for a specific customer. Use when a customer is being harmed. **Level 5: Segment-Specific** Disables AI for a customer segment (e.g., vulnerable customers). Use for targeted protection. ### Implementation Pattern ``` Kill Switch Service (Independent Infrastructure) ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ Admin │───▶│ Kill SW │───▶│ Agent │ │ Console │ │ Service │ │ Services │ └─────────────┘ └─────────────┘ └─────────────┘ │ ┌──────┴──────┐ │ State │ │ Store │ └─────────────┘ Key properties: - Kill switch service on separate infrastructure - Agents poll kill switch state (not push) - Polling interval < 30 seconds - Default to disabled if can't reach kill switch service - State store is highly available ``` ### Kill Switch Testing Quarterly drills are minimum. Test: - Activation time (must be < 60 seconds) - All levels work correctly - Authorisation works - Audit trail is generated - Recovery after reactivation Document results. Address any failures immediately. ## Circuit Breaker Pattern Circuit breakers provide automatic protection without human intervention. When metrics degrade beyond thresholds, the breaker trips and the system fails safe. ### Three States **Closed** (Normal Operation) Requests flow through. Failures are counted. **Open** (Fail Fast) Requests immediately fail or use fallback. No calls to degraded system. **Half-Open** (Testing Recovery) Limited requests test if system has recovered. Success → Closed. Failure → Open. ### Trigger Thresholds Define thresholds based on your risk appetite: | Metric | Warning | Trip | |--------|---------|------| | Error rate | >2% | >5% | | P99 latency | >10s | >15s | | Consecutive failures | 5 | 10 | | Cost rate | >150% baseline | >200% baseline | Thresholds should be based on data from normal operation, not guesses. ### Implementation ```python class AICircuitBreaker: def __init__(self, name: str, config: BreakerConfig): self.name = name self.state = BreakerState.CLOSED self.failure_count = 0 self.last_failure_time = None self.config = config def call(self, operation: Callable) -> Result: if self.state == BreakerState.OPEN: if self._should_test_recovery(): self.state = BreakerState.HALF_OPEN else: return self._fallback() try: result = operation() self._record_success() return result except Exception as e: self._record_failure() if self._should_trip(): self._trip() raise def _trip(self): self.state = BreakerState.OPEN self._alert_operations() self._log_trip() ``` ### Fallback Behaviour When the circuit breaker is open, what happens? **Option 1: Graceful Degradation** Fall back to simpler processing (see below). **Option 2: Queue for Later** Store request for processing when system recovers. **Option 3: Human Handoff** Route to human agent immediately. **Option 4: Inform and Wait** Tell customer there's a delay, try again later. Choose based on use case. Payment processing can't queue. FAQ answers can. ## Graceful Degradation Framework When AI isn't available, what happens? Define levels in advance. ### Four Degradation Levels **Level 1: Full Service** AI operating normally. All features available. **Level 2: Degraded Service** AI available but reduced capability. Some features disabled. Latency may be higher. Example: Complex queries disabled, simple queries still work. **Level 3: Fallback Service** AI unavailable. Alternative service path. Human backup. Example: Route to human agents. Use rule-based system. **Level 4: Offline** Service unavailable. Clear messaging to customers. Example: "This service is temporarily unavailable. Please call us." ### Recovery Time Objectives Define how quickly you must recover at each level: | Scenario | Detection | Failover | Full Recovery | |----------|-----------|----------|---------------| | LLM provider issue | 1 min | 2 min | Provider dependent | | Single agent service | 30 sec | 1 min | 5 min | | Kill switch activation | N/A | <1 min | N/A | | Full platform | 2 min | 5 min | 30 min | Test these. Document actual performance. Improve. ### Degradation Triggers What triggers each level? **Level 2 triggers**: - LLM latency > 5 seconds - Error rate > 2% - Single feature failing **Level 3 triggers**: - LLM unavailable - Error rate > 10% - Kill switch activated (feature level) **Level 4 triggers**: - Platform-wide failure - Global kill switch - Security incident ### Customer Communication Each level needs prepared customer messaging: **Level 2**: "We're experiencing slower than usual response times." **Level 3**: "Our AI assistant is temporarily unavailable. I'm connecting you with a team member." **Level 4**: "This service is temporarily unavailable. Please call [number] or try again later." Prepare these in advance. Don't write crisis messaging during a crisis. ## Control Testing Controls that aren't tested don't work. Establish a testing regime: ### Continuous (Every Deployment) - Input validation tests - Policy enforcement tests - Output filtering tests - Unit tests for all control code ### Periodic (Quarterly) - Kill switch drills - Circuit breaker testing - Penetration testing - Adversarial prompt testing - Full control inventory review ### Annual - Independent control audit - Regulatory compliance review - Full disaster recovery test - Third-party security assessment ### Testing Evidence Document: - What was tested - How it was tested - Results (pass/fail) - Issues found - Remediation actions - Sign-off This evidence is what auditors and regulators want to see. ## Common Control Failures ### Failure 1: Single Point of Control One control that "should catch everything." It won't. Defence in depth requires multiple independent controls. ### Failure 2: Controls in AI System Kill switch controlled by the AI it's meant to stop. Circuit breaker logic in the service it's meant to protect. Controls must be independent. ### Failure 3: Untested Controls "We have a kill switch" but it's never been tested. When you need it, it won't work. Test quarterly. ### Failure 4: Manual-Only Response All controls require human intervention. At 3am on a bank holiday, no one is watching. Automatic controls provide first response. ### Failure 5: No Fallback Defined Circuit breaker trips and... then what? Define fallback behaviour before you need it. ## When to Seek Expert Help Defence in depth for AI requires getting the architecture right. External expertise helps when: - **Designing control frameworks**: Start with a proven structure - **Implementing kill switches**: Independence and reliability are critical - **Testing controls**: Adversarial testing requires specialist skills - **Responding to incidents**: Rapid, appropriate response limits damage I help regulated firms design and implement defence in depth for AI systems. [Get in touch →](/contact/) ## Related Reading - [AI Agent Governance for Financial Services](/writings/ai-governance-financial-services/) - Governance framework - [LLM Provider Risk Management](/writings/llm-provider-risk-management/) - Third-party risk - [The Substrate Pattern](/writings/ai-agent-substrate-pattern/) - Execution envelopes for agents --- *Dipankar Sarkar is a technology advisor specializing in AI safety for regulated industries. He has designed control frameworks for AI systems at scale and helps financial services firms build defence in depth that satisfies regulators while enabling innovation. [Learn more →](/about/)* --- ## AI Agent Governance for Financial Services: The Complete Framework URL: https://www.dipankar.name/writings/ai-governance-financial-services/ Date: 2025-11-18 Tags: ai-governance, financial-services, regulatory-compliance, ai-risk-management, banking, consumer-duty, sm-cr Dipankar Sarkar explains AI agent governance for banks and financial services. The Tiered Governance Model and Three Lines of Defence framework for regulatory compliance with SS1/23, Consumer Duty, and SM&CR. Banks deploying AI agents face a governance challenge: how to capture the efficiency gains while satisfying regulators, protecting customers, and maintaining accountability. The good news is that existing UK regulatory frameworks—SS1/23 for model risk, Consumer Duty for customer outcomes, SM&CR for personal accountability—already provide comprehensive coverage. There is no regulatory vacuum. The challenge is applying these frameworks correctly. This guide presents a governance framework for AI agents in regulated financial services, built on two core models: the **Tiered Governance Model** for proportionate controls and the **Three Lines of Defence** adapted for AI systems. **Central thesis**: Governance designed in from day one enables innovation. Governance retrofitted after deployment fails. ## Why Existing Regulation Applies Some firms wait for AI-specific regulation. This is a mistake. UK regulators have been clear: they regulate firms, not technologies. Existing frameworks apply. ### SS1/23: Model Risk Management The PRA's Supervisory Statement 1/23 applies to AI agents as models. It requires: - **Governance**: Board-level oversight, clear ownership, defined risk appetite - **Validation**: Independent testing before deployment - **Monitoring**: Ongoing performance and drift detection - **Documentation**: Model specification, limitations, intended use AI agents are models. SS1/23 applies. ### Consumer Duty The FCA's Consumer Duty requires firms to deliver good outcomes for retail customers. For AI agents, this means: - **Avoid foreseeable harm**: Test for bias, errors, and edge cases before deployment - **Support customer understanding**: Explain AI involvement when relevant - **Price and value**: AI efficiency gains should benefit customers, not just margins - **Consumer support**: Escalation paths when AI can't help An AI agent that harms customers violates Consumer Duty regardless of whether the harm was intended. ### SM&CR: Personal Accountability The Senior Managers and Certification Regime requires named individuals accountable for firm activities. For AI agents: - **Named SMF**: Every AI agent must have a named Senior Manager accountable for its outcomes - **Reasonable steps**: The SMF must demonstrate reasonable steps to prevent harm - **Evidence**: Documentation proving oversight, not just assertions AI cannot be accountable. A human must be. SM&CR makes this explicit and personal. ## The Tiered Governance Model Not all AI agents are equal. A chatbot answering FAQs poses different risks than an agent making credit decisions. Governance should be proportionate. ### Tier 1: High Autonomy, High Impact **Characteristics**: - Autonomous decisions affecting customers or finances - Significant potential for harm if wrong - Regulatory or reputational consequences **Examples**: Credit decisioning, fraud detection with automatic blocks, investment recommendations **Governance Requirements**: - Board-level reporting (quarterly minimum) - Independent validation before deployment - Real-time monitoring with human oversight - Kill switch with <60 second activation - Named SMF with documented accountability ### Tier 2: Moderate Autonomy or Impact **Characteristics**: - Decisions require human approval, OR - Impact is moderate, OR - Established use case with known risks **Examples**: Customer service with human escalation, document processing with review, risk scoring for human decision **Governance Requirements**: - Senior management oversight - Periodic validation (annual minimum) - Standard monitoring and alerting - Documented escalation procedures - Business owner accountability ### Tier 3: Low Autonomy, Low Impact **Characteristics**: - Informational only, no decisions - Minimal customer or financial impact - Well-understood, stable use case **Examples**: Internal knowledge search, document summarisation, meeting notes **Governance Requirements**: - Business owner sign-off - Self-assessment against standards - Standard IT controls - Proportionate monitoring ### Tier Assignment Process Tier assignment should happen early—Week 2 of a project, not Week 12. Criteria: | Factor | Tier 1 | Tier 2 | Tier 3 | |--------|--------|--------|--------| | **Autonomy** | Fully autonomous | Human approval required | Informational only | | **Customer impact** | Direct, significant | Indirect or moderate | Minimal | | **Financial impact** | >£X per decision | £Y-X per decision | <£Y per decision | | **Regulatory exposure** | High (credit, AML) | Moderate | Low | | **Reversibility** | Difficult to reverse | Reversible with effort | Easily reversible | Early tier assignment drives the right governance intensity from the start. ## Three Lines of Defence for AI The Three Lines model is standard in financial services. Applied to AI: ### First Line: Business and Technology **Responsibilities**: - Owns the AI agent and its outcomes - Implements controls and monitoring - Provides frontline risk management - Escalates issues promptly **For AI specifically**: - Defines use case and constraints - Implements input/output controls - Monitors performance metrics - Manages day-to-day operations ### Second Line: Risk and Compliance **Responsibilities**: - Sets standards and frameworks - Provides independent challenge - Monitors first line effectiveness - Reports to governance forums **For AI specifically**: - Defines AI risk appetite and policy - Reviews tier assignments - Validates control effectiveness - Monitors regulatory developments ### Third Line: Internal Audit **Responsibilities**: - Independent assurance - Tests control design and operation - Reports to Audit Committee **For AI specifically**: - Audits AI governance framework effectiveness - Tests AI-specific controls (kill switches, bias detection) - Validates documentation completeness - Assesses regulatory compliance ### AI Governance Forum A cross-functional forum provides oversight across all three lines: **Composition**: - CRO (Chair) - Business representatives - Technology leadership - Risk and Compliance - Legal and Data Protection **Responsibilities**: - Approves Tier 1 deployments - Reviews Tier 2 deployments - Sets AI risk appetite - Monitors aggregate AI risk - Escalates to Board **Cadence**: Monthly, with emergency convening capability ## Evidence by Design Regulators want evidence, not assertions. "We review AI outputs carefully" is an assertion. Logs showing every review, reviewer, and outcome are evidence. ### Documentation Requirements For SS1/23 compliance, each AI agent needs: **Model Specification**: - Purpose and intended use - Architecture and LLM providers - Data inputs and outputs - Known limitations **Risk Assessment**: - Tier classification with rationale - Risk and control mapping - Residual risk acceptance **Validation Report**: - Testing methodology - Results and findings - Limitations identified **Monitoring Specification**: - Metrics tracked - Thresholds and alerts - Escalation procedures ### Audit Trail Architecture Build evidence generation into the system: ``` Every interaction logged: - Timestamp - Customer identifier (pseudonymised) - Input received - Processing steps - LLM calls and responses - Output delivered - Latency and cost ``` This isn't just compliance—it's debugging, performance management, and customer service. But compliance requires it. ### Control Testing Evidence Controls must be: - **Defined**: Clear specification exists - **Implemented**: Actually built and deployed - **Enabled**: Switched on in production - **Tested**: Verified to work - **Monitored**: Continuous operation confirmed - **Auditable**: Evidence available on request Every control needs evidence for each criterion. ## Human Accountability Is Non-Negotiable AI cannot be accountable under any current framework. When an AI agent causes harm: - The firm is liable - A named individual is accountable - "The AI did it" is not a defence ### Meaningful Human Oversight Human-in-the-loop must be meaningful, not rubber-stamping: **Meaningful**: Human reviews transaction details, applies judgment, can reject or modify **Rubber-stamp**: Human clicks "approve" on a queue they can't practically review Regulators distinguish between these. So should you. ### The Reasonable Steps Defence Under SM&CR, the "reasonable steps" defence requires demonstrating: - Appropriate governance was in place - Controls were designed and operated effectively - Issues were escalated and addressed - Documentation supports the narrative This defence requires evidence. Build the evidence generation into the system. ## Engage Governance Early The biggest governance mistake is engaging late. Week 12 discovery that a Tier 1 deployment needs Board approval delays launch by months. ### Week 2, Not Week 12 Early engagement means: - Tier assignment in project planning - Governance requirements in project scope - Compliance resource allocation from start - No surprises at deployment ### Governance as Enabler Counterintuitively, early governance engagement accelerates deployment: - Requirements are clear from the start - Documentation is built as you go - Validation is planned into timeline - Approval is fast because reviewers are prepared Governance surprises slow you down. Planned governance speeds you up. ## Common Governance Failures ### Failure 1: Treating AI as Exempt "It's just a chatbot" doesn't exempt it from Consumer Duty. If it interacts with customers, Consumer Duty applies. ### Failure 2: Governance Theatre Forms without substance. Review meetings that don't review. Documentation that no one reads. Regulators see through this. ### Failure 3: Late Engagement Discovering governance requirements at deployment. Building evidence after the fact. Explaining to the Board why launch is delayed. ### Failure 4: Unclear Accountability Multiple owners means no owner. "The team" isn't accountable under SM&CR. A named individual is. ### Failure 5: Static Governance Governance set at deployment and never revisited. LLM providers change. Use cases evolve. Governance must adapt. ## When to Seek Expert Help AI governance in financial services requires regulatory expertise, technical depth, and practical experience. External expertise helps when: - **Starting AI agent deployment**: Getting governance right from the start prevents expensive rework - **Preparing for regulatory scrutiny**: Supervisors are increasingly focused on AI—be ready - **Scaling AI initiatives**: Governance that works for one agent may not work for twenty - **Responding to incidents**: AI failures require rapid, appropriate response I help regulated firms implement AI governance frameworks that satisfy regulators while enabling innovation. [Get in touch →](/contact/) ## Related Reading - [Defence in Depth for AI Agents](/writings/defence-in-depth-ai-agents/) - Kill switches, circuit breakers, and control layers - [LLM Provider Risk Management](/writings/llm-provider-risk-management/) - Third-party risk for AI systems - [AI Agent Safety: The Substrate Pattern](/writings/ai-agent-substrate-pattern/) - Architectural patterns for AI safety --- *Dipankar Sarkar is a technology advisor specializing in AI governance for regulated industries. With experience building AI systems at scale and deep knowledge of UK financial services regulation, he helps banks and insurers deploy AI agents that satisfy regulators while delivering business value. [Learn more →](/about/)* --- ## Hiring Engineers Who Ship: Building High-Performance Teams from Scratch URL: https://www.dipankar.name/writings/hiring-engineers-who-ship/ Date: 2025-11-15 Tags: engineering-hiring, team-building, engineering-leadership, technical-interviews, startup-scaling, engineering-culture Dipankar Sarkar shares the hiring framework he used to build engineering teams at Nykaa, Hike, and Orangewood Labs. How to identify engineers who ship production systems, not just pass interviews. The hardest problem in engineering leadership isn't architecture or technology choices. It's hiring. Every system I've built—from Nykaa's e-commerce platform serving millions to Orangewood's robotics SDK—was ultimately shaped by who was on the team. Get hiring right and most other problems become manageable. Get it wrong and no amount of process can compensate. After scaling teams from 5 to 50+ engineers across four companies, I've developed a framework that prioritizes shipping ability over interview performance. It's not perfect, but it consistently identifies engineers who deliver. ## The Interview-Shipping Gap Traditional technical interviews measure a narrow slice of engineering ability: algorithm knowledge, system design theory, and communication under pressure. These correlate weakly with what actually matters in production environments. Engineers who ship well share traits that interviews rarely test: - **Scope management**: They instinctively reduce scope to hit deadlines without sacrificing quality on what remains. - **Debugging intuition**: They navigate unfamiliar codebases and find root causes faster than they solve LeetCode problems. - **Technical judgment**: They know when to build, when to buy, and when to defer a decision. - **Collaborative momentum**: They make the people around them faster, not just themselves. The gap between interview performance and shipping ability is the central problem of engineering hiring. Every element of my framework attempts to close this gap. ## The Three-Signal Framework I evaluate candidates on three signals, each designed to predict shipping ability rather than interview performance. ### Signal 1: Production Scars Every engineer who has shipped real systems carries scars—war stories about outages, migrations gone wrong, performance crises, and deadline pressure. These scars are impossible to fake and reveal more about engineering ability than any whiteboard exercise. **What I ask**: - "Tell me about a production incident where you were the primary debugger. Walk me through your process." - "Describe a project where you had to cut scope mid-sprint. What did you cut and why?" - "What's the worst technical decision you've made? How did you discover it was wrong?" **What I listen for**: - Specificity. Real incidents have timestamps, metric values, and customer impact numbers. Fabricated ones are vague. - Ownership. Engineers who ship say "I decided" and "I missed." Engineers who don't say "we" for failures and "I" for successes. - Learning loops. The best engineers have changed their approach based on past failures. They can articulate what they do differently now. At Nykaa, I hired an engineer whose only notable credential was maintaining a high-traffic WordPress plugin. But his description of debugging a race condition in the plugin's caching layer—complete with the specific MySQL query that was locking—told me more about his ability than any system design question could. ### Signal 2: Technical Taste Technical taste is the ability to distinguish between solutions that are merely correct and solutions that are appropriate. It's the difference between an engineer who can build anything and one who builds the right thing. **How I test it**: I present a real problem from our codebase—not a sanitized interview question, but an actual design decision we faced. I describe the constraints and ask for their approach. I'm not looking for the "right" answer. I'm looking for: - **Constraint awareness**: Do they ask about scale, timeline, team size, and maintenance burden before proposing a solution? - **Trade-off articulation**: Can they describe what they're giving up with their approach, not just what they're gaining? - **Appropriate complexity**: Do they reach for the simplest solution that works, or do they over-engineer for hypothetical future requirements? At Hike, I gave candidates a real problem: we needed to serve personalized content feeds to 30 million users with sub-200ms latency. The best candidates didn't jump to architecture diagrams. They asked about access patterns, tolerance for staleness, and whether 200ms was a p50 or p99 target. The questions revealed more than the answers. ### Signal 3: Collaborative Evidence Shipping is a team sport. Individual brilliance that doesn't compose with other engineers' work is net negative at scale. **What I look for**: - **Code review history**: If available, their PR reviews tell me everything. Do they catch real issues or nitpick style? Do they suggest alternatives or just point out problems? - **Open source contributions**: Not the quantity—the quality of interaction. How do they respond to feedback on their PRs? How do they review others'? - **Reference patterns**: I ask references a single question: "Would you want to work with this person again on a hard deadline?" The hesitation or enthusiasm in the answer is the signal. ## The Anti-Patterns Patterns I've learned to avoid through expensive mistakes: ### The Brilliant Loner High individual output, zero collaborative impact. They write impressive code that nobody else can maintain. They solve hard problems but create harder ones for the team. I hired two of these early in my career. Both produced exceptional individual work. Both left behind codebases that required rewrites after they left. **Detection**: Ask about their most recent team project. If every story centers on their individual contribution with teammates as supporting characters, that's the pattern. ### The Resume Architect They've worked at impressive companies on impressive-sounding projects. But when you dig into their specific contributions, the specificity evaporates. "I was part of the team that built X" without being able to describe their particular decisions, trade-offs, or mistakes. **Detection**: Ask "What was your most controversial technical decision on that project?" Engineers who actually made decisions have answers. Engineers who were adjacent to decisions don't. ### The Perpetual Optimizer They can improve any system's performance by 20% but can't ship a new feature end-to-end. They gravitate toward optimization because it's measurable and safe. New features require product judgment and tolerance for ambiguity. **Detection**: Ask "Describe something you built from zero—from blank file to production." If they struggle or pivot to optimization stories, that's the pattern. ## Structuring the Interview Process Based on these signals, here's how I structure the process: ### Stage 1: Production Scar Screen (30 minutes, remote) A single interviewer has a conversation focused entirely on past production experience. No coding. No system design. Just stories about real work. **Pass criteria**: At least two detailed, specific stories about shipping under constraints. The stories should include concrete decisions, measurable outcomes, and lessons learned. This stage filters out roughly 60% of candidates—not because they're bad engineers, but because they haven't yet accumulated the production experience our roles require. ### Stage 2: Technical Taste Exercise (60 minutes, remote or on-site) Present a real problem from your domain. Give the candidate 10 minutes to read the context, then 50 minutes of collaborative discussion. **Pass criteria**: The candidate asks good questions before proposing solutions, articulates trade-offs without prompting, and arrives at an approach appropriate for the stated constraints (not an impressive approach—an appropriate one). ### Stage 3: Pair Programming on Real Code (90 minutes, on-site or screen-share) The candidate works on a real task from your backlog with a team member. Not a toy problem—an actual task that would take an experienced engineer 2-4 hours, scoped to 90 minutes. **Pass criteria**: The candidate makes meaningful progress, communicates their thought process, asks good questions about the codebase, and handles ambiguity without freezing. ### Stage 4: Team Interaction (60 minutes) The candidate meets 3-4 team members in informal settings. This isn't a technical evaluation—it's a collaboration evaluation. Each team member answers one question afterward: "Would you want to pair with this person on a hard problem?" ## Calibration and Iteration Every hire is a hypothesis. Validate it: - **90-day review**: Compare the candidate's actual performance against the signals that led to the hire decision. Which signals were predictive? Which were misleading? - **False negative tracking**: When possible, track candidates you rejected who were hired elsewhere. Did they succeed? This is harder to measure but invaluable for calibrating your standards. - **Team feedback loops**: After each hiring round, debrief with the interview team. Not just "should we hire this person" but "did our process surface the right information?" At Nykaa, this calibration process led us to drop algorithm questions entirely after we found zero correlation between algorithm performance and first-quarter shipping output. The time we freed up went into longer pair programming sessions, which proved far more predictive. ## Scaling the Framework This framework works at every scale I've operated at, but the emphasis shifts: **5-person team**: Over-index on Signal 1 (production scars) and Signal 3 (collaboration). You need people who can ship independently and won't create friction in a small team. Technical taste matters less because you can course-correct in real time. **15-person team**: Signal 2 (technical taste) becomes critical. You can no longer review every decision. You need engineers whose judgment you trust when you're not in the room. **50-person organization**: All three signals matter equally, but you also need to train other interviewers to evaluate them. The framework must be teachable, not just intuitive. ## The Uncomfortable Truth The best hiring framework still has a significant error rate. You will hire people who don't work out. You will reject people who would have been excellent. The goal isn't perfection—it's a better hit rate than the industry standard, which is remarkably low. What separates good engineering organizations from great ones isn't just who they hire—it's how quickly they recognize and correct hiring mistakes, and how consistently they learn from both successes and failures in their process. Every engineer on your team either accelerates or decelerates the whole. Hiring is the highest-leverage activity in engineering leadership. Treat it that way. --- ## Scaling AI-Assisted Development: From Startup to Enterprise URL: https://www.dipankar.name/writings/scaling-ai-development-teams/ Date: 2025-10-22 Tags: ai-assisted-development, engineering-leadership, team-scaling, enterprise-ai, devops, software-reliability Dipankar Sarkar explains how to scale AI-assisted development from small teams to enterprise. The Pivot Point framework for knowing when informal practices break. AI-assisted development scales poorly without deliberate architecture. Solo developers enjoy pure productivity gains. Small teams experience friction but manage. Larger organizations discover that the practices enabling individual speed create organizational chaos. This is the scaling challenge: the same AI capabilities that accelerate individuals can decelerate organizations when applied without structure. Understanding where the breakpoints occur—and what to do at each transition—is essential for engineering leaders navigating AI adoption. ## The Pivot Point Framework Every scaling system has pivot points—thresholds where existing approaches stop working and new patterns become necessary. For AI-assisted development, three pivot points dominate: ### Pivot Point 1: Solo to Team (2-5 developers) **What breaks**: Implicit context sharing. Solo developers hold everything in their heads. They know what the AI generated, why they accepted it, and what constraints apply. When a second developer joins, this knowledge doesn't transfer automatically. **Symptoms**: - "Why is this code structured this way?" (Lost generation context) - "This doesn't match our patterns" (Inconsistent AI outputs) - "I changed X and Y broke" (Hidden dependencies in AI-generated code) **What to add**: - Version control for prompts and generation context - Code review with generation context visible - Shared prompt libraries and templates - Documentation requirements for AI-generated components ### Pivot Point 2: Team to Teams (5-20 developers) **What breaks**: Informal coordination. A single team can align through conversation. Multiple teams can't. AI-generated code from one team may violate invariants another team depends on. Without formal contracts, integration becomes a constant negotiation. **Symptoms**: - "Our service broke when they deployed" (Interface contract violations) - "We can't update because they depend on our implementation details" (Coupling through AI-generated code) - "Nobody knows if this invariant is real or accidental" (Lost specification intent) **What to add**: - Explicit interface contracts between teams - Automated contract testing in CI - Central registry of system-wide invariants - Cross-team review for AI-generated shared components ### Pivot Point 3: Teams to Organization (20+ developers) **What breaks**: Cultural enforcement. Small organizations can enforce norms through osmosis. Large organizations can't. "We review AI-generated code carefully" becomes "we say we review AI-generated code carefully" becomes "our review standards vary by team and deadline pressure." **Symptoms**: - "Different teams have completely different AI practices" (Standards drift) - "We don't know what's AI-generated vs hand-written" (Auditability gaps) - "Compliance is asking questions we can't answer" (Regulatory risk) **What to add**: - Mechanical enforcement of AI development standards (CI gates, not guidelines) - Centralized AI usage tracking and analytics - Formal training and certification programs - Compliance-ready audit trails ## Organizational Patterns for AI-Native Development ### Pattern 1: The AI Enablement Team A cross-functional team responsible for: - Developing shared AI tooling and integrations - Maintaining prompt libraries and templates - Establishing and evolving AI development standards - Training other teams on effective practices - Monitoring organization-wide AI usage and outcomes **When to create**: At Pivot Point 2, when multiple teams need coordination. **Anti-pattern to avoid**: The AI Enablement Team becomes a bottleneck. Their job is to enable, not to gatekeep. ### Pattern 2: The Three-Layer Review Not all AI-generated code deserves equal scrutiny. Differentiate: **Layer 1 (Automated)**: Linting, formatting, basic security scanning. Applies to all code. Fast, cheap, mechanical. **Layer 2 (Team Review)**: Standard code review with generation context visible. Reviewers understand what was generated and why. **Layer 3 (Architecture Review)**: For changes that affect system boundaries, invariants, or cross-team contracts. Slower, more expensive, but necessary for high-impact changes. Match review depth to blast radius. Quick utility functions get Layer 1. New service architectures get Layer 3. ### Pattern 3: The Specification Repository Maintain a central repository of: - System-wide invariants with enforcement mechanisms - Cross-team interface contracts - Canonical prompts for common tasks - Prohibited patterns and why they're prohibited - Decision records for AI architecture choices This repository becomes the source of truth for what AI-generated code must respect. Teams reference it. CI enforces it. New team members learn from it. ### Pattern 4: The Observability Stack You can't manage what you can't see. Track: **Generation Metrics**: - What AI tools are being used? - How much code is AI-generated vs hand-written? - What's the acceptance rate for AI suggestions? **Quality Metrics**: - Defect rates in AI-generated vs hand-written code - Time to resolve AI-generated code issues - Review rejection rates for AI-generated code **Compliance Metrics**: - Audit trail completeness - Policy violation incidents - Training completion rates These metrics inform where to invest in tooling, training, and process improvement. ## Metrics That Matter Traditional engineering metrics (lines of code, features shipped, velocity) become misleading with AI assistance. Lines of code measures nothing when AI generates thousands in minutes. Features shipped says nothing about maintainability. ### Better Metrics for AI-Native Development **Change Failure Rate**: What percentage of changes result in degraded service or require rollback? This measures the quality of AI-generated code in production. **Mean Time to Recovery**: When AI-generated code fails, how quickly can teams diagnose and fix it? This measures whether teams understand what they deployed. **Invariant Violation Rate**: How often do systems violate declared invariants? This measures specification completeness. **Generation Context Retention**: What percentage of AI-generated code has accessible generation context 30/60/90 days later? This measures auditability. **Review Depth Score**: Are high-impact changes getting appropriate review? This measures process compliance. These metrics (inspired by DORA research) measure outcomes that matter: reliability, recoverability, maintainability. ## The Compliance Dimension Regulated industries face additional scaling challenges. AI-assisted development introduces questions regulators are only beginning to ask: **Auditability**: Can you demonstrate what code was AI-generated and what the generation context was? **Determinism**: Can you reproduce a deployment given the same inputs? (Note: LLMs are non-deterministic.) **Accountability**: Who is responsible for AI-generated code that causes harm? **Explainability**: Can you explain why the code does what it does, even if AI generated it? ### Compliance Patterns **Generation Logging**: Every AI code generation is logged with timestamp, model version, prompt, and output. **Human-in-the-Loop**: All AI-generated code requires human review before deployment. Review is logged. **Change Attribution**: Every change is attributed to a responsible human, even if AI assisted. **Model Inventory**: Track which AI models were used for what, with version history. These patterns add overhead but are necessary for regulated industries and increasingly expected for enterprise deployments. ## Transition Strategies Moving an organization through pivot points requires deliberate transition management: ### Strategy 1: Incremental Adoption Start with low-risk applications of AI assistance. Build organizational muscle. Expand gradually. **Phase 1**: AI for test generation, documentation, boilerplate **Phase 2**: AI for feature implementation with review **Phase 3**: AI for architecture exploration with oversight **Phase 4**: AI agents with substrate constraints Each phase builds capabilities needed for the next. ### Strategy 2: Parallel Systems Run AI-assisted and traditional development in parallel. Compare outcomes. Let evidence guide adoption. This is expensive but reduces risk. Use for organizations where AI adoption failures would be catastrophic. ### Strategy 3: Team Pilots Select willing teams to pioneer AI practices. Document what works and what doesn't. Scale successful patterns. Choose pilot teams carefully: they need both enthusiasm and discipline. Pure enthusiasm produces hype. Pure discipline produces rejection. ## Common Scaling Mistakes **Mistake 1: Assuming small-team practices scale** They don't. What works for 3 developers fails for 30. Plan for transitions before they're forced. **Mistake 2: Governance as afterthought** Adding governance to an already-chaotic AI deployment is harder than building it in. Start with structure. **Mistake 3: Metrics without action** Measuring AI usage is easy. Acting on what measurements reveal is hard. Don't collect metrics you won't use. **Mistake 4: Training as one-time event** AI capabilities evolve continuously. Training must too. Build ongoing learning into team routines. **Mistake 5: Central mandates without local buy-in** Mandating AI practices without developer input produces compliance theater. Involve teams in developing standards. ## When to Seek Expert Help Scaling AI-assisted development is a significant organizational change. External expertise helps when: - **Approaching a pivot point**: Getting the transition right matters more than getting there first - **Experiencing scaling pains**: Symptoms are present but root causes are unclear - **Facing regulatory scrutiny**: Compliance requirements are evolving and complex - **Building governance frameworks**: Starting with good structure is easier than fixing bad structure I help engineering organizations navigate AI adoption at scale through advisory engagements, organizational assessments, and transformation programs. [Get in touch →](/contact/) ## Related Reading - [Production-Ready AI-Assisted Development](/writings/ai-assisted-development-production-guide/) - Foundation framework - [AI Agent Safety: The Substrate Pattern](/writings/ai-agent-substrate-pattern/) - Agent architecture - [Invariants in AI-Generated Code](/writings/invariants-ai-generated-code/) - Specification patterns --- *Dipankar Sarkar is a technology advisor with 18+ years of experience scaling engineering organizations. He has led teams from startup to enterprise scale and helps organizations adopt AI-assisted development without sacrificing reliability or velocity. [Learn more →](/about/)* --- ## AI Agent Safety: The Substrate Pattern for LLM-Powered Systems URL: https://www.dipankar.name/writings/ai-agent-substrate-pattern/ Date: 2025-09-15 Tags: ai-agents, llm, ai-safety, substrate-pattern, production-systems, guardrails, agentic-ai Dipankar Sarkar explains the Substrate Pattern for AI agent safety. How to build execution envelopes that constrain LLM-powered systems while preserving their capabilities. AI agents—systems where LLMs take autonomous actions—are the frontier of AI-assisted development. They promise to move beyond code generation to task completion: deploy this service, fix this bug, respond to this incident. But they also introduce risks that code generation doesn't: a code generator produces text; an agent takes actions with real-world consequences. The fundamental problem: **AI agents are non-deterministic**. The same prompt may produce different outputs. The same context may lead to different actions. And unlike traditional software, you can't unit test every possible behavior because the behavior space is unbounded. This article presents the **Substrate Pattern**—an architectural approach that separates AI proposals from deterministic execution, providing the safety guarantees production systems require while preserving the capabilities that make agents valuable. ## The Agent Fallacy The Agent Fallacy is the belief that AI agents can be trusted to self-constrain. It manifests in several forms: **"The model will follow instructions"**: LLMs are trained on human data. They exhibit the full range of human behaviors, including ignoring instructions, misinterpreting context, and producing confident nonsense. Instructions are probabilistic guidance, not deterministic constraints. **"We'll prompt engineer the risks away"**: Prompt engineering is valuable but insufficient. No prompt can anticipate every context. No instruction set can prevent every failure mode. The attack surface is the entire space of possible inputs. **"The agent will ask before doing anything dangerous"**: This assumes the agent can identify danger, that its judgment aligns with yours, and that it will consistently choose to ask rather than act. None of these are guaranteed. Lisanne Bainbridge's "Ironies of Automation" (1983) applies perfectly: the more autonomous we make systems, the more critical human oversight becomes, yet the harder that oversight is to provide. AI agents amplify this irony. ## The Substrate Pattern The Substrate Pattern addresses the Agent Fallacy by separating concerns: **The Agent** (non-deterministic): Proposes actions based on intent and context. Can be creative, exploratory, and unpredictable. This is where LLM capabilities shine. **The Substrate** (deterministic): Evaluates proposals against constraints. Executes permitted actions through controlled pathways. Logs everything. This is traditional software engineering. The agent proposes. The substrate disposes. ### Core Architecture ``` ┌─────────────────────────────────────────────────────┐ │ SUBSTRATE │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ Permission │ │ Execution │ │ Audit │ │ │ │ Boundary │→→│ Engine │→→│ Log │ │ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ ↑ │ │ │ Proposals │ │ ┌─────────────────────────────────────────────────┤ │ │ AGENT │ │ │ Intent → Context → Reasoning → Proposed Action │ │ └─────────────────────────────────────────────────┘ └─────────────────────────────────────────────────────┘ ``` ### Permission Boundaries The permission boundary defines what actions are possible, not just what actions are intended. This is the key insight: **constraint enforcement must be mechanical, not cultural**. **Input Boundaries**: What information can the agent access? Scope data access to the minimum required. Don't give a deployment agent access to production databases if it only needs deployment configurations. **Output Boundaries**: What actions can the agent take? Enumerate permitted actions explicitly. If an action isn't on the list, it can't happen—regardless of what the agent proposes. **Resource Boundaries**: What resources can the agent consume? Limit API calls, compute time, and cost. Unbounded agents become expensive agents. **Blast Radius**: If the agent fails catastrophically, what's the worst outcome? Design the substrate so the worst case is acceptable. ### Execution Engine The execution engine translates validated proposals into actions. It provides: **Deterministic Execution**: Same proposal, same action. The execution engine is traditional software—testable, predictable, auditable. **Idempotency**: Actions can be safely retried. If the agent proposes the same thing twice, the system handles it gracefully. **Rollback Capability**: Actions can be reversed. This is critical for recovery from agent errors that pass permission checks but produce undesired outcomes. **Rate Limiting**: Actions are throttled. An agent in a bad loop can't execute unlimited actions before humans notice. ### Audit Log Every proposal is logged. Every execution is logged. Every rejection is logged. The audit log provides: **Forensics**: When something goes wrong, you can reconstruct exactly what happened—the proposal, the context, the decision, the outcome. **Learning**: Rejected proposals reveal what the agent is trying to do that it shouldn't. Accepted proposals that led to bad outcomes reveal gaps in permission boundaries. **Compliance**: For regulated systems, the audit log provides the evidence trail that manual oversight can't. ## Implementation Patterns ### Pattern 1: Action Enumeration Don't allow arbitrary actions. Enumerate them. ```python PERMITTED_ACTIONS = { "deploy_staging": DeployStagingAction, "deploy_production": DeployProductionAction, "rollback": RollbackAction, "scale_up": ScaleUpAction, "scale_down": ScaleDownAction, } def execute_proposal(proposal: AgentProposal) -> Result: if proposal.action not in PERMITTED_ACTIONS: log_rejection(proposal, "action_not_permitted") return Rejection("Action not in permitted set") action_class = PERMITTED_ACTIONS[proposal.action] action = action_class(proposal.parameters) if not action.validate(): log_rejection(proposal, "validation_failed") return Rejection("Action parameters invalid") result = action.execute() log_execution(proposal, result) return result ``` The agent can propose anything. Only enumerated actions execute. ### Pattern 2: Graduated Permissions Not all actions are equal. Dangerous actions require more validation. **Tier 1 (Automatic)**: Read-only actions, reversible changes, sandbox operations. Execute immediately after basic validation. **Tier 2 (Confirmed)**: Production changes, resource allocation, external API calls. Require explicit confirmation or cooldown period. **Tier 3 (Supervised)**: Destructive operations, security-sensitive actions, compliance-relevant changes. Require human approval before execution. The permission tier is determined by the action, not the agent's confidence. High confidence from an LLM is not the same as low risk. ### Pattern 3: Capability Tokens Grant capabilities explicitly, not implicitly. ```python class AgentCapabilities: def __init__(self, token: CapabilityToken): self.can_read_staging = token.has_capability("read:staging") self.can_write_staging = token.has_capability("write:staging") self.can_read_production = token.has_capability("read:production") self.can_write_production = token.has_capability("write:production") def create_deployment_agent() -> Agent: # Deployment agent can write staging, read production token = CapabilityToken([ "read:staging", "write:staging", "read:production" # Note: no write:production ]) return Agent(capabilities=AgentCapabilities(token)) ``` Capabilities are granted at agent creation, not inferred from context. The agent can't acquire capabilities it wasn't given. ### Pattern 4: Circuit Breakers Stop runaway agents automatically. ```python class AgentCircuitBreaker: def __init__(self, agent_id: str): self.failure_count = 0 self.failure_threshold = 5 self.reset_timeout = timedelta(minutes=15) self.last_failure = None def record_failure(self): self.failure_count += 1 self.last_failure = datetime.now() if self.failure_count >= self.failure_threshold: self.trip_breaker() def trip_breaker(self): notify_operators(f"Agent {self.agent_id} circuit breaker tripped") disable_agent(self.agent_id) ``` When agents fail repeatedly, stop them. Investigate before resuming. ## Common Mistakes in Agent Architecture **Mistake 1: Trust by default** Agents should have zero capabilities until explicitly granted. Never start with full access and try to restrict—start with no access and carefully expand. **Mistake 2: Logging as afterthought** The audit log is not optional. Design it first. An agent without comprehensive logging is an agent you can't debug, can't audit, and can't trust. **Mistake 3: Assuming rollback is possible** Some actions are irreversible—sent emails, deleted data, deployed contracts. Design permission boundaries with irreversibility in mind. **Mistake 4: Human review as security theater** If humans must approve every action, agents provide no value. If humans never review anything, agents are unsupervised. Find the right level of oversight for your risk profile. ## When to Seek Expert Help Agent architecture requires getting safety right the first time. Organizations often benefit from external expertise when: - **Deploying production agents for the first time**: The patterns that work in demos fail at scale - **Operating in regulated environments**: Compliance requirements for autonomous systems are evolving and complex - **Experiencing agent incidents**: A misbehaving agent is a sign of architectural gaps - **Scaling agent deployments**: More agents means more attack surface I help engineering teams design and implement the Substrate Pattern through architecture reviews, security assessments, and implementation guidance. [Get in touch →](/contact/) ## Related Reading - [Production-Ready AI-Assisted Development](/writings/ai-assisted-development-production-guide/) - The Vibes Inside Guardrails framework - [Invariants in AI-Generated Code](/writings/invariants-ai-generated-code/) - What AI can't infer - [Scaling AI-Assisted Development](/writings/scaling-ai-development-teams/) - Organizational patterns --- *Dipankar Sarkar is a technology advisor specializing in AI-native development and production systems. He has architected ML systems serving millions of users and helps organizations build safe, reliable AI agent infrastructure. [Learn more →](/about/)* --- ## Invariants in AI-Generated Code: What LLMs Can't Infer URL: https://www.dipankar.name/writings/invariants-ai-generated-code/ Date: 2025-08-28 Tags: invariants, ai-code-generation, software-reliability, design-by-contract, ai-code-quality, production-systems Dipankar Sarkar explains why unwritten invariants are the primary source of AI-generated defects. How to make implicit constraints explicit for production-ready AI-assisted development. The most insidious bugs in AI-generated code aren't syntax errors or logic mistakes. They're violations of invariants that were never written down—constraints that exist in the developer's mind, in the codebase's history, in the organization's implicit knowledge, but nowhere the AI could access. An invariant is a property that must always be true. "Account balances never go negative." "A user session is always associated with a valid user." "Timestamps are monotonically increasing." When invariants hold, systems behave predictably. When they're violated, systems fail in ways that are hard to diagnose and expensive to fix. AI can't infer invariants you haven't specified. This is the fundamental gap in AI-assisted development, and closing it is the key to production-ready AI-generated code. ## The Unwritten Invariant Problem Consider a prompt: "Generate a function to transfer money between accounts." The AI produces something like: ```python def transfer(from_account, to_account, amount): from_account.balance -= amount to_account.balance += amount ``` This code is correct according to the prompt. It's also dangerously incomplete: - **What if `amount` is negative?** The transfer becomes a theft. - **What if `from_account.balance < amount`?** The balance goes negative. - **What if the second operation fails?** The money disappears. - **What if `from_account == to_account`?** The balance is corrupted. These aren't edge cases—they're invariants. The real specification includes: - `amount > 0` (precondition) - `from_account.balance >= amount` (precondition) - `from_account != to_account` (precondition) - `sum(all_balances)` is unchanged (invariant) - Either both operations succeed or neither does (atomicity invariant) The AI didn't know these because you didn't say them. And you didn't say them because they felt obvious. This gap—between what's obvious to humans and what's specified to AI—is where production failures hide. ## Dijkstra's Insight: Invariants as the Real Specification Edsger Dijkstra, in "A Discipline of Programming" (1976), argued that the core of correct programs is not the code but the invariants the code maintains. The code is merely a mechanism for transitioning between states while preserving what must remain true. This insight transforms how we think about AI code generation: **Traditional view**: The code is the specification. Tests verify the code. **Dijkstra's view**: The invariants are the specification. Code implements the invariants. Tests verify that invariants hold. **Applied to AI**: Specify the invariants. Let AI generate code. Verify that the code maintains the invariants. When invariants are correct and the code maintains them, many implementations become acceptable—including AI-generated ones. When invariants are unstated, even "correct" code can be wrong. ## Design by Contract for AI-Assisted Development Bertrand Meyer's Design by Contract (1988) provides the vocabulary for making invariants explicit: **Preconditions**: What must be true before a function runs. These are the function's requirements of its callers. **Postconditions**: What will be true after a function runs. These are the function's promises to its callers. **Class/Module Invariants**: What must be true whenever an object is in a stable state (between method calls). Applied to the transfer example: ```python def transfer(from_account, to_account, amount): """ Transfer money between accounts. Preconditions: - amount > 0 - from_account.balance >= amount - from_account != to_account - both accounts are active Postconditions: - from_account.balance == old(from_account.balance) - amount - to_account.balance == old(to_account.balance) + amount - sum of all balances unchanged Invariants: - no balance is ever negative - all operations are atomic (full success or full rollback) """ # Implementation here ``` With this specification, AI can generate implementation *and* you have a contract to verify against. ## Encoding Invariants at Multiple Levels Invariants should be enforced mechanically, not just documented. Defense in depth: ### Level 1: Type System Make illegal states unrepresentable. ```python from typing import NewType from dataclasses import dataclass # Amount can't be negative by construction PositiveAmount = NewType('PositiveAmount', Decimal) def make_positive_amount(value: Decimal) -> PositiveAmount | None: if value <= 0: return None return PositiveAmount(value) @dataclass(frozen=True) class Transfer: from_account: AccountId to_account: AccountId amount: PositiveAmount def __post_init__(self): if self.from_account == self.to_account: raise ValueError("Cannot transfer to same account") ``` The type system encodes constraints. AI-generated code using these types inherits the constraints automatically. ### Level 2: Database Constraints Enforce invariants at the persistence layer. ```sql CREATE TABLE accounts ( id UUID PRIMARY KEY, balance DECIMAL NOT NULL, CONSTRAINT positive_balance CHECK (balance >= 0) ); CREATE TABLE transfers ( id UUID PRIMARY KEY, from_account UUID REFERENCES accounts(id), to_account UUID REFERENCES accounts(id), amount DECIMAL NOT NULL, CONSTRAINT positive_amount CHECK (amount > 0), CONSTRAINT different_accounts CHECK (from_account != to_account) ); ``` Even if AI-generated application code violates invariants, the database rejects the transaction. ### Level 3: Runtime Assertions Check invariants at system boundaries. ```python def transfer(from_account: Account, to_account: Account, amount: Decimal): # Precondition checks assert amount > 0, f"Amount must be positive, got {amount}" assert from_account.balance >= amount, "Insufficient funds" assert from_account.id != to_account.id, "Cannot transfer to same account" initial_sum = from_account.balance + to_account.balance # Implementation from_account.balance -= amount to_account.balance += amount # Invariant check assert from_account.balance >= 0, "Balance went negative" assert from_account.balance + to_account.balance == initial_sum, "Money created or destroyed" ``` Assertions catch violations at runtime. In production, they trigger alerts. In development, they fail fast. ### Level 4: Property-Based Tests Verify invariants hold for generated inputs. ```python from hypothesis import given, strategies as st @given( balance1=st.decimals(min_value=0, max_value=10000), balance2=st.decimals(min_value=0, max_value=10000), amount=st.decimals(min_value=Decimal('0.01'), max_value=10000) ) def test_transfer_preserves_total(balance1, balance2, amount): if amount > balance1: return # Skip invalid inputs acc1 = Account(balance=balance1) acc2 = Account(balance=balance2) initial_total = acc1.balance + acc2.balance transfer(acc1, acc2, amount) assert acc1.balance + acc2.balance == initial_total assert acc1.balance >= 0 assert acc2.balance >= 0 ``` Property-based tests check invariants across a wide range of inputs, including edge cases AI might generate but humans wouldn't test. ## Common Invariant Categories ### Consistency Invariants - Foreign key relationships are valid - Aggregate values match their components (sum of line items == order total) - Cross-field constraints hold (end_date > start_date) ### Safety Invariants - Security-sensitive values are never logged - Authentication state is always valid - Permissions are checked before actions ### Ordering Invariants - Events are processed in causal order - Versions are monotonically increasing - State machine transitions are valid ### Resource Invariants - Allocated resources are eventually released - Connection pools don't leak - File handles are closed Each category represents implicit knowledge that AI can't access unless you make it explicit. ## Making Invariants AI-Accessible For AI to generate invariant-respecting code, invariants must be in context: ### Approach 1: Invariant Docstrings Include invariants in the modules AI will reference: ```python """ Account Module INVARIANTS: - Account.balance >= 0 always - Sum of all account balances equals initial system balance - All balance-changing operations are atomic - Account.status must be ACTIVE for any balance operations PRECONDITIONS for balance operations: - User must be authenticated - User must have permission for the account - Amount must be positive """ ``` ### Approach 2: Contract Files Maintain explicit contract specifications AI can reference: ```yaml # contracts/accounts.yaml module: accounts invariants: - name: positive_balance description: Account balance is never negative expression: "account.balance >= 0" enforcement: - database_constraint - runtime_assertion - name: balance_conservation description: Total system balance is constant expression: "sum(all_balances) == INITIAL_SYSTEM_BALANCE" enforcement: - audit_log_verification ``` ### Approach 3: Typed Interfaces Use types that encode constraints: ```python class AccountService(Protocol): def transfer( self, from_account: ActiveAccount, # Type enforces account is active to_account: ActiveAccount, amount: PositiveDecimal, # Type enforces positive ) -> TransferResult: """ Returns: TransferResult with either Success or InsufficientFunds Never raises. Never returns partial success. """ ... ``` The types themselves communicate invariants that AI can understand and respect. ## When to Seek Expert Help Identifying and encoding invariants is skilled work. Organizations benefit from external expertise when: - **Implicit knowledge is everywhere**: Long-lived codebases accumulate invariants that no one remembers documenting - **AI-generated code is failing in production**: Invariant violations often manifest as mysterious bugs - **Building new AI-assisted workflows**: Getting invariants right upfront is easier than fixing them later - **Preparing for compliance audits**: Regulators want to see how constraints are enforced I help engineering teams identify hidden invariants, encode them mechanically, and build AI-assisted workflows that respect them. [Get in touch →](/contact/) ## Related Reading - [Production-Ready AI-Assisted Development](/writings/ai-assisted-development-production-guide/) - The Vibes Inside Guardrails framework - [AI Agent Safety: The Substrate Pattern](/writings/ai-agent-substrate-pattern/) - Execution envelopes for agents - [Scaling AI-Assisted Development](/writings/scaling-ai-development-teams/) - Team patterns --- *Dipankar Sarkar is a technology advisor specializing in AI-native development and production systems. He has built ML systems serving hundreds of millions of users and helps organizations make implicit knowledge explicit for AI-assisted development. [Learn more →](/about/)* --- ## Production-Ready AI-Assisted Development: The Vibes Inside Guardrails Framework URL: https://www.dipankar.name/writings/ai-assisted-development-production-guide/ Date: 2025-07-14 Tags: ai-assisted-development, ai-code-generation, production-systems, software-reliability, guardrails, llm, engineering-leadership Dipankar Sarkar explains how to use AI code generation in production systems. The Vibes Inside Guardrails framework for capturing AI productivity while maintaining reliability. The productivity gains from AI-assisted development are real. Engineers report 2-3x faster initial implementation. Prototypes that once took weeks now take hours. But there's a hidden cost: systems built fast often fail slow, with breakdowns emerging months after creation when the original context is gone and the AI conversation that generated the code has long been forgotten. This guide presents a framework for capturing AI's genuine benefits while adding the constraints production systems require. The core insight: **the future is not a choice between AI speed and engineering discipline. It is AI speed *inside* engineering discipline.** ## The Productivity-Reliability Paradox AI code generation works by compressing intent into implementation. You describe what you want; the model produces code. This compression is powerful—it eliminates boilerplate, handles syntax, and accelerates initial delivery. But compression has costs: **The Theory-Building Problem**: Peter Naur observed in 1985 that programming is fundamentally about building mental models—theories of how systems work. When AI generates code you didn't struggle to write, you may not build the theory needed to maintain it. The code works, but your understanding is shallow. **Temporal Asymmetry**: Creation takes days or months. Operation spans years. Evolution is continuous. AI accelerates creation but doesn't change the other timelines. Barry Boehm's research shows 60-80% of software costs occur *after* initial development—in exactly the phases AI doesn't help with. **The Convincing Local Maximum**: AI-generated code often works immediately. Tests pass. Users are happy. This immediate success obscures distant failure modes that emerge only under production pressure—edge cases the AI couldn't anticipate, state mutations it didn't model, integration failures it couldn't predict. ## The Failure Curve: Why AI Code Breaks Later Lehman's Laws of Software Evolution, established in 1980, state that software must continuously adapt or become progressively less useful. AI-generated systems are not exempt. **Immediate failures** (within days) are easy: wrong logic, missing edge cases, obvious bugs. Teams catch these in testing. **Delayed failures** (weeks to months) are harder: performance degradation under load, memory leaks over time, state corruption from concurrent access, integration drift as dependencies update. **Far-future failures** (months to years) are hardest: the original AI conversation is gone, the context that informed generation is lost, and the code resists modification because no one built the theory of how it works. The failure curve for AI-generated systems is often *inverted*: low initial failure rates (the code works!) that increase over time as maintenance pressure accumulates against shallow understanding. ## The Vibes Inside Guardrails Framework The solution is not to abandon AI assistance but to constrain it within mechanical boundaries. This is the **Vibes Inside Guardrails** paradigm: **Vibes**: The exploratory, creative, fast iteration that AI enables. Express intent. Generate code. Evaluate results. Iterate quickly. This is where AI shines. **Guardrails**: Mechanical constraints that the AI-generated code must satisfy. Contracts, invariants, type systems, automated verification. These are enforced by systems, not culture. The key insight: **freedom and discipline are not opposites**. Discipline enables freedom by providing the boundaries within which creativity is safe. ### Implementation Pattern: Sandbox + Ledger **Sandbox**: The environment where AI-assisted iteration happens. Generate, experiment, refine. Low friction, high speed. **Ledger**: The audit trail that makes AI decisions accountable. Every generation is logged. Every deployment is traced. Intent is preserved as a versioned artifact, not lost in ephemeral conversations. The sandbox provides freedom. The ledger provides accountability. Together, they capture AI's benefits while adding production discipline. ## The Three Pillars of AI-Native Production Systems ### 1. Intent as Contract Natural language intent is ambiguous. Contracts are precise. When you tell an AI "build a user authentication system," the intent is clear to you but underdetermined for production. What are the preconditions? What guarantees does the system provide? What invariants must hold? **Design by Contract** (Bertrand Meyer, 1988) provides the answer: translate intent into preconditions (what must be true before), postconditions (what will be true after), and invariants (what remains true always). AI can generate implementations. You must specify contracts. ### 2. Invariants Over Implementations An invariant is a property that must always be true. "Account balances never go negative." "User sessions are always associated with valid users." "Timestamps are always monotonically increasing." Tests check specific cases. Invariants constrain all cases. When invariants are correct, many implementations are acceptable—including AI-generated ones. The primary source of AI-generated defects is *unwritten invariants*—constraints that exist in the developer's mind but were never made explicit. AI can't infer what you haven't specified. [Read more about invariants in AI-generated code →](/writings/invariants-ai-generated-code/) ### 3. State as First-Class Concern Software is not just code. It is code + data + state + history. AI generates code. It doesn't model your state machines, understand your data migrations, or anticipate your operational patterns. Leslie Lamport's work on distributed systems shows that production reliability is primarily about state reliability—and state is exactly what AI code generation ignores. Make state explicit. Model it formally. Test state transitions, not just functions. ## Implementation Patterns ### Pattern 1: Contract-First Generation Before using AI to generate implementation: 1. Write the contract: preconditions, postconditions, invariants 2. Generate implementation against the contract 3. Verify generated code satisfies the contract 4. Preserve both contract and generation context The contract becomes the specification. The AI becomes an implementation engine. The verification becomes mechanical. ### Pattern 2: Invariant Encoding Encode invariants at multiple levels: - **Database constraints**: Foreign keys, check constraints, unique indexes - **Type systems**: Make illegal states unrepresentable - **Runtime assertions**: Check invariants at boundaries - **Property-based tests**: Verify invariants hold for generated inputs Each layer catches failures the others miss. Defense in depth applies to AI-generated code too. ### Pattern 3: Event Sourcing for Auditability Prompts are ephemeral by default. Make them durable. Preserve the prompt that generated each component. Log the model version, the context window, the generation parameters. When code fails in production, you need to reconstruct the intent that created it. Event sourcing extends beyond prompts to all system state changes. The log becomes the source of truth. Every mutation is traceable. [Read more about the Substrate Pattern for AI agents →](/writings/ai-agent-substrate-pattern/) ## Common Mistakes **Mistake 1: Trusting AI output without verification** AI generates plausible code that compiles and passes basic tests. This is not the same as correct code. Verify against contracts, not just syntax. **Mistake 2: Losing generation context** Six months from now, when the code breaks, you'll need to understand why it was generated this way. Preserve the conversation, the prompt evolution, the rejected alternatives. **Mistake 3: Skipping the theory-building** Fast generation is not a license to skip understanding. If you can't explain why the code works, you can't safely modify it. Invest in building the mental model, even for AI-generated code. **Mistake 4: Treating guardrails as optional** Cultural norms don't scale. Process compliance degrades under pressure. Mechanical enforcement—type systems, CI checks, automated verification—is the only reliable guardrail. ## When to Seek Expert Help Organizations often benefit from external expertise when: - **Adopting AI-assisted development at scale**: The patterns that work for solo developers break for teams - **Building in regulated industries**: Compliance requires audit trails and verification that AI workflows don't naturally provide - **Experiencing the failure curve**: Systems built fast are now failing slow, and the team lacks context to fix them - **Transitioning team skills**: Engineers need to shift from implementation to specification, evaluation, and operation I help engineering teams implement the Vibes Inside Guardrails framework through advisory engagements, architecture reviews, and organizational transformation programs. [Get in touch →](/contact/) ## Related Reading - [AI Agent Safety: The Substrate Pattern](/writings/ai-agent-substrate-pattern/) - Execution envelopes for LLM-powered systems - [Invariants in AI-Generated Code](/writings/invariants-ai-generated-code/) - What AI can't infer and how to make it explicit - [Scaling AI-Assisted Development](/writings/scaling-ai-development-teams/) - From startup to enterprise patterns --- *Dipankar Sarkar is a technology advisor specializing in AI-native development and production systems. With 18+ years building platforms at scale and 55+ patents filed, he helps organizations capture AI's productivity benefits while maintaining production reliability. [Learn more →](/about/)* --- ## Optimizing ARM Builds in CI/CD: Emulation, Native Builds, and Cloud-Native Solutions URL: https://www.dipankar.name/writings/optimizing-arm-builds-cicd-pipelines-challenges-solutions/ Date: 2024-10-21 Tags: devops, ci-cd, arm-architecture, cloud-infrastructure, performance-optimization Practical strategies for optimizing ARM builds in CI/CD pipelines. Emulation challenges, native build setup, cross-compilation, and cloud-native approaches with benchmarks. # Optimizing ARM Builds in CI/CD Pipelines: Challenges and Solutions In the ever-evolving landscape of software development and deployment, the rise of ARM-based architectures has introduced new challenges to Continuous Integration and Continuous Deployment (CI/CD) pipelines. This article delves into the intricacies of ARM builds, exploring the bottlenecks encountered in CI/CD environments and presenting cutting-edge solutions to enhance performance and efficiency. ## Table of Contents 1. [Introduction: The ARM Revolution in Computing](#introduction-the-arm-revolution-in-computing) 2. [Understanding the Challenges of ARM Builds in CI/CD](#understanding-the-challenges-of-arm-builds-in-cicd) 3. [The QEMU Conundrum: Emulation vs. Native Performance](#the-qemu-conundrum-emulation-vs-native-performance) 4. [Strategies for Optimizing ARM Builds](#strategies-for-optimizing-arm-builds) 5. [Advanced Techniques: Distributed and Cloud-Native Approaches](#advanced-techniques-distributed-and-cloud-native-approaches) 6. [Case Studies: Real-World Implementations](#case-studies-real-world-implementations) 7. [Future Trends: The Evolving Landscape of ARM in CI/CD](#future-trends-the-evolving-landscape-of-arm-in-cicd) 8. [Conclusion: Embracing ARM in Modern CI/CD Pipelines](#conclusion-embracing-arm-in-modern-cicd-pipelines) ## Introduction: The ARM Revolution in Computing The advent of ARM (Advanced RISC Machine) architecture has ushered in a new era of computing, promising enhanced power efficiency and performance across a wide range of devices. From smartphones to data centers, ARM-based processors are becoming increasingly prevalent, necessitating adaptations in software development and deployment practices. ## Understanding the Challenges of ARM Builds in CI/CD Integrating ARM builds into existing CI/CD pipelines presents several unique challenges: 1. **Architectural Differences**: The fundamental differences between x86 and ARM architectures require careful consideration in build processes. 2. **Cross-Compilation Complexity**: Developing on x86 machines for ARM targets introduces additional layers of complexity. 3. **Performance Bottlenecks**: Emulation-based builds can significantly increase build times, impacting CI/CD efficiency. 4. **Toolchain Compatibility**: Ensuring compatibility between build tools and ARM architecture can be a non-trivial task. ## The QEMU Conundrum: Emulation vs. Native Performance QEMU (Quick Emulator) has been a go-to solution for ARM builds on x86 machines, but it comes with significant drawbacks: ### Advantages of QEMU - **Flexibility**: Allows ARM builds on widely available x86 infrastructure. - **Compatibility**: Supports a wide range of ARM architectures and versions. ### Disadvantages of QEMU - **Performance Overhead**: Emulation introduces substantial slowdowns, often 5-10 times slower than native execution. - **Resource Intensity**: QEMU-based builds consume significantly more CPU and memory resources. - **Debugging Challenges**: Troubleshooting issues in emulated environments can be complex and time-consuming. ## Strategies for Optimizing ARM Builds To address the challenges of ARM builds in CI/CD pipelines, consider implementing the following strategies: 1. **Native ARM CI Runners**: - Utilize physical ARM hardware or ARM-based cloud instances for CI/CD tasks. - Pros: Maximum performance, accurate representation of target environment. - Cons: Potentially higher infrastructure costs, limited availability. 2. **Hybrid Build Approaches**: - Combine x86 and ARM runners, offloading ARM-specific tasks to native hardware. - Pros: Balanced performance and resource utilization. - Cons: Increased complexity in pipeline management. 3. **Optimized Cross-Compilation**: - Leverage advanced cross-compilation techniques and optimized toolchains. - Pros: Improved build times on x86 hardware. - Cons: Requires expertise in cross-compilation intricacies. 4. **Containerized Build Environments**: - Utilize Docker's multi-architecture support for consistent build environments. - Pros: Enhanced portability and reproducibility. - Cons: Potential performance overhead in emulated scenarios. ## Advanced Techniques: Distributed and Cloud-Native Approaches For organizations dealing with large-scale ARM builds, consider these advanced techniques: 1. **Distributed Build Systems**: - Implement tools like Bazel or BuildGrid to parallelize and distribute build tasks. - Leverage cloud resources for on-demand scaling of build capacity. 2. **Cloud-Native ARM Solutions**: - Explore emerging cloud services offering native ARM compute instances. - Utilize serverless architectures for flexible and scalable build processes. 3. **AI-Driven Build Optimization**: - Employ machine learning algorithms to predict and optimize build configurations. - Dynamically allocate resources based on historical build data and current pipeline needs. ## Case Studies: Real-World Implementations ### Case Study 1: Tech Giant X's ARM Transition Tech Giant X successfully transitioned their mobile app CI/CD pipeline to support ARM builds by: - Implementing a hybrid build system with both x86 and ARM runners. - Utilizing advanced caching mechanisms to reduce redundant build steps. - Adopting a microservices architecture to parallelize independent build components. Result: 40% reduction in overall build times and improved resource utilization. ### Case Study 2: Startup Y's Cloud-Native Approach Startup Y leveraged cloud-native technologies to optimize their ARM build pipeline: - Implemented serverless build functions on ARM-based cloud instances. - Utilized container-based builds with multi-architecture support. - Employed AI-driven resource allocation to optimize cloud spending. Result: 60% cost reduction and 3x improvement in build throughput. ## Future Trends: The Evolving Landscape of ARM in CI/CD As ARM continues to gain traction, we can expect several trends to shape the future of CI/CD: 1. **Increased Native ARM CI/CD Tools**: More tools and platforms will offer native ARM support, reducing reliance on emulation. 2. **Advanced Emulation Technologies**: New emulation techniques may bridge the performance gap between x86 and ARM builds. 3. **Edge Computing Integration**: ARM's efficiency will drive more edge-based CI/CD workflows, especially in IoT scenarios. 4. **Cross-Platform Development Frameworks**: Emergence of frameworks designed to streamline development across x86 and ARM architectures. ## Related Reading Interested in more DevOps and platform engineering content? - [EdgeML and the Future of Robotics](/writings/edgeml-future-robotics-next-generation-sdk-platform/) - Building scalable platforms with edge computing - [Building Real-Time Data Ingestion Framework](/writings/building-realtime-data-ingestion-analytics-framework-ecommerce/) - Event-driven architecture and AWS Lambda at scale ## Conclusion: Embracing ARM in Modern CI/CD Pipelines The integration of ARM builds into CI/CD pipelines represents both a challenge and an opportunity for organizations. By understanding the unique characteristics of ARM architecture and implementing optimized build strategies, teams can harness the full potential of ARM while maintaining efficient and scalable development workflows. As the computing landscape continues to evolve, embracing ARM in CI/CD processes will become increasingly crucial for staying competitive in the fast-paced world of software development and deployment. --- *About the author: [Dipankar Sarkar](/about/) is a technology leader specializing in AI, robotics, and cloud architecture. With 18+ years of experience building scalable platforms and CI/CD pipelines, he has led infrastructure projects at Orangewood Labs and major e-commerce companies. [View all posts](/writings/) | [Get in touch](/contact/)* --- ## EdgeML and the Future of Robotics: Building the Next-Generation SDK and Platform URL: https://www.dipankar.name/writings/edgeml-future-robotics-next-generation-sdk-platform/ Date: 2024-03-15 Tags: edge-ai, robotics, machine-learning, iot, platform-architecture Dipankar Sarkar built a next-generation robotics SDK at Orangewood Labs using EdgeML for on-device AI processing, enabling intelligent robotic systems. I'm thrilled to share insights into one of our most ambitious projects at Orangewood Labs: the development of a next-generation SDK and platform for robotics, powered by EdgeML. This initiative is set to redefine how we approach robot programming and management, bringing unprecedented levels of intelligence and efficiency to robotic systems. ## The EdgeML Revolution in Robotics Edge Machine Learning, or EdgeML, is transforming the landscape of robotics by enabling AI processing directly on robotic devices, rather than relying solely on cloud-based solutions. This paradigm shift brings several key advantages: 1. **Reduced Latency**: Critical for real-time decision making in robotics. 2. **Enhanced Privacy**: Sensitive data can be processed locally, reducing security risks. 3. **Offline Capabilities**: Robots can function intelligently even without constant internet connectivity. 4. **Bandwidth Efficiency**: Only relevant data needs to be transmitted to the cloud. ## Our Vision: A Unified Robotics Platform Our goal is to create a comprehensive SDK and platform that leverages the power of EdgeML to simplify robot programming, enhance capabilities, and improve interoperability. Here's what we're building: ### 1. Modular SDK - **Language Agnostic**: Support for multiple programming languages (Python, C++, Rust) to cater to diverse developer preferences. - **Hardware Abstraction Layer**: Enabling code portability across different robotic hardware. - **EdgeML Integration**: Built-in support for deploying and running machine learning models on robotic edge devices. ### 2. Intuitive Development Environment - **Visual Programming Interface**: Drag-and-drop tools for non-programmers to create simple robotic behaviors. - **Advanced IDE Integration**: Plugins for popular IDEs to support professional developers. - **Simulation Environment**: For testing and debugging robotic applications before deployment. ### 3. Robust Management Platform - **Fleet Management**: Tools for monitoring and managing multiple robots in real-time. - **Over-the-Air Updates**: Seamless deployment of software updates and new ML models. - **Performance Analytics**: Detailed insights into robot performance and health. ### 4. Interoperability Focus - **Open Standards**: Adherence to and promotion of open robotics standards. - **API-First Approach**: Comprehensive APIs for integration with external systems and services. - **Plugin Architecture**: Allowing easy extension of platform capabilities. ## Collaboration with Industry Leaders Our development efforts are strengthened through strategic partnerships: - **Viam**: Collaborating on advanced robotics control systems. - **Freedom Robotics**: Enhancing our fleet management capabilities. - **Solomon3D**: Improving our simulation and visualization tools. - **Cogniteam** and **Piknik**: Working on advanced AI and cognitive computing integration. ## Technical Challenges and Innovations Developing this platform presents several unique challenges: 1. **Heterogeneous Hardware Support**: Creating a unified interface for vastly different robotic systems. - Solution: Developing a sophisticated hardware abstraction layer and leveraging containerization technologies. 2. **Efficient EdgeML Deployment**: Optimizing ML models for resource-constrained edge devices. - Solution: Implementing model compression techniques and developing custom EdgeML runtimes. 3. **Real-Time Distributed Computing**: Enabling seamless cooperation between multiple robots. - Solution: Developing a custom distributed computing framework optimized for robotic applications. 4. **Security and Privacy**: Ensuring robust security in a distributed edge computing environment. - Solution: Implementing end-to-end encryption, secure enclaves for sensitive computations, and blockchain-based audit trails. ## The Road Ahead As we continue to develop this platform, we're excited about several future enhancements: 1. **Federated Learning Integration**: Enabling robots to collectively learn and improve without sharing raw data. 2. **Quantum-Inspired Algorithms**: Exploring quantum computing principles to solve complex optimization problems in robotics. 3. **Augmented Reality Integration**: Developing tools for AR-assisted robot programming and monitoring, complementing our [RoboGPT natural language interface](/writings/robogpt-natural-language-robot-programming-industry-transformation/). 4. **Bio-Inspired Computing**: Incorporating principles from neuroscience to create more adaptive robotic behaviors. ## Related Reading Explore more about our robotics innovations: - [RoboGPT: Natural Language Robot Programming](/writings/robogpt-natural-language-robot-programming-industry-transformation/) - Making robot programming accessible through natural language and transforming industries - [AutoInspect and AutoSpray: ML-Driven Precision](/writings/autoinspect-autospray-ml-driven-precision-industrial-robotics/) - Industrial applications of our ML-powered robotics ## Conclusion: Shaping the Future of Robotics Our SDK and platform represent more than just a set of tools; they're a vision for the future of robotics. By leveraging EdgeML and creating a unified, intelligent platform, we're paving the way for a new generation of robots that are more capable, efficient, and easier to program and manage. This initiative has the potential to democratize robotics development, accelerate innovation, and open up new possibilities across industries. From manufacturing and healthcare to exploration and environmental conservation, the applications are boundless. At Orangewood Labs, we're committed to pushing the boundaries of what's possible in robotics. As we continue to refine and expand our SDK and platform, we invite developers, researchers, and industry partners to join us in shaping the future of this exciting field. Stay tuned for more updates as we work towards launching this groundbreaking platform and ushering in a new era of intelligent, edge-powered robotics! --- *About the author: [Dipankar Sarkar](/about/) is a technology leader with expertise in AI, robotics, and edge computing. As former Head of AI & Platform at Orangewood Labs, he architected next-generation robotics platforms powered by EdgeML. [View all posts](/writings/) | [Get in touch](/contact/)* --- ## AutoInspect and AutoSpray: ML-Driven Precision in Industrial Robotics URL: https://www.dipankar.name/writings/autoinspect-autospray-ml-driven-precision-industrial-robotics/ Date: 2024-01-20 Tags: computer-vision, machine-learning, industrial-automation, robotics, deep-learning Dipankar Sarkar built AutoInspect and AutoSpray at Orangewood Labs - ML and computer vision systems revolutionizing industrial quality control and precision spray painting. As we enter 2024, I'm excited to share the remarkable progress we've made at Orangewood Labs with our AutoInspect and AutoSpray solutions. These innovative systems represent a significant leap forward in applying machine learning and computer vision to industrial robotics, particularly in the realms of quality control and precision manufacturing. ## The Challenge: Precision and Consistency in Industrial Processes In many industries, inspection and spray painting tasks require a level of precision and consistency that can be challenging for human workers to maintain over long periods. Traditional automated solutions often lack the flexibility to adapt to varying conditions or product specifications. Our goal with AutoInspect and AutoSpray was to create systems that combine the precision of robotics with the adaptability of advanced machine learning. ## AutoInspect: Revolutionizing Quality Control AutoInspect is our cutting-edge solution for automated visual inspection: 1. **Advanced Computer Vision**: Utilizes state-of-the-art deep learning models for image analysis. 2. **Multi-Spectrum Imaging**: Incorporates various imaging technologies (visible light, infrared, UV) for comprehensive inspection. 3. **Real-Time Defect Detection**: Identifies and classifies defects with high accuracy in real-time. 4. **Adaptive Learning**: Continuously improves its detection capabilities based on new data. 5. **Integration with Production Lines**: Seamlessly integrates with existing manufacturing processes for immediate feedback and action. ## AutoSpray: Precision Coating with AI AutoSpray brings a new level of sophistication to industrial spray painting: 1. **3D Surface Mapping**: Uses advanced sensors to create detailed 3D maps of objects for optimal spray coverage. 2. **Dynamic Path Planning**: AI algorithms calculate the most efficient spray paths in real-time. 3. **Environmental Adaptation**: Adjusts spray parameters based on environmental conditions like temperature and humidity. 4. **Consistent Finish Quality**: Ensures uniform coating thickness and appearance across complex geometries. 5. **Material Efficiency**: Minimizes overspray and waste, reducing material costs and environmental impact. ## The Power of Machine Learning in Industrial Applications Both AutoInspect and AutoSpray leverage cutting-edge machine learning techniques: 1. **Deep Learning for Vision**: Convolutional Neural Networks (CNNs) power our image analysis capabilities. 2. **Reinforcement Learning**: Used in AutoSpray for optimizing spray patterns and paths. 3. **Transfer Learning**: Allows rapid adaptation to new products or materials with minimal additional training. 4. **Anomaly Detection**: Advanced algorithms identify unusual patterns or defects that might escape traditional inspection methods. ## Real-World Impact and Industry Interest The response from our industry partners has been overwhelmingly positive: - **Automotive Industry**: Major car manufacturers are using AutoSpray for more efficient and consistent paint application. - **Electronics Manufacturing**: AutoInspect is being employed for quality control in smartphone and computer component production. - **Aerospace**: Both systems are being tested for use in aircraft component manufacturing and maintenance. ## Challenges and Solutions Developing these systems came with its share of challenges: 1. **Data Diversity**: We created synthetic datasets and employed data augmentation techniques to train our models on a wide range of scenarios. 2. **Real-Time Processing**: Optimized our algorithms and leveraged edge computing to achieve the necessary speed for real-time operation. 3. **Integration with Legacy Systems**: Developed flexible interfaces to ensure compatibility with existing industrial equipment. ## The Road Ahead As we continue to refine AutoInspect and AutoSpray, we're exploring several exciting avenues: 1. **Generative AI for Defect Simulation**: Using GANs to generate synthetic defect images for more robust training. 2. **Collaborative Robotics**: Integrating these systems with cobots for safer human-robot collaboration in quality control and finishing processes. Our [RoboGPT natural language interface](/writings/robogpt-natural-language-robot-programming-industry-transformation/) makes it easier than ever to program and control these collaborative systems. 3. **Predictive Maintenance**: Extending AutoInspect's capabilities to predict potential equipment failures before they occur. 4. **Sustainable Coating Technologies**: Developing AutoSpray variants for new, environmentally friendly coating materials. ## Related Reading Interested in more robotics and AI innovations? Check out: - [RoboGPT: Natural Language Robot Programming](/writings/robogpt-natural-language-robot-programming-industry-transformation/) - How we're making robot programming as easy as having a conversation and transforming industries - [EdgeML and the Future of Robotics](/writings/edgeml-future-robotics-next-generation-sdk-platform/) - Building intelligent, edge-powered robotic systems ## Conclusion: Shaping the Future of Industrial Processes AutoInspect and AutoSpray represent more than just technological advancements; they're ushering in a new era of smart manufacturing. By combining the precision of robotics with the adaptability of AI, we're enabling industries to achieve levels of quality, efficiency, and consistency that were previously unattainable. As we move forward, we're excited to continue pushing the boundaries of what's possible in industrial automation. The future of manufacturing is intelligent, adaptive, and precise – and at Orangewood Labs, we're proud to be leading the way. Stay tuned for more innovations as we continue to revolutionize the world of industrial robotics! --- *About the author: [Dipankar Sarkar](/about/) is a technology leader specializing in AI, robotics, and industrial automation. As former Head of AI & Platform at Orangewood Labs, he developed cutting-edge ML solutions for industrial robotics. [View all posts](/writings/) | [Get in touch](/contact/)* --- ## Machine Learning at Hike: Five AI Systems Serving Millions URL: https://www.dipankar.name/writings/machine-learning-hike-ai-systems/ Date: 2024-01-20 Tags: machine-learning, computer-vision, nlp, recommendation-systems, federated-learning, tensorflow, python Dipankar Sarkar led development of five ML systems at Hike serving millions: computer vision avatars, vernacular NLP, social matchmaking, trust & safety, and gaming matchmaking. As the leader of the Machine Learning team at Hike Limited, I spearheaded the development of five major AI-driven features that transformed user experience across the platform. These projects spanned computer vision, natural language processing, social matching, trust & safety, and gaming - collectively serving millions of users across India. ## 1. Hikemoji: Computer Vision for Avatar Generation ### Project Overview Hikemoji generated personalized avatars directly from users' selfies. My role focused on developing sophisticated computer vision models to match avatar components to specific facial attributes. ###Technical Approach **Core Technologies:** - Python, TensorFlow, PyTorch, OpenCV - BigQuery for data storage - Airflow for workflow orchestration **Key Components:** 1. **Facial Feature Extraction**: Models to identify and map key facial features from selfies 2. **Component Matching Algorithm**: AI-driven system matching facial features with avatar components 3. **Style Transfer Techniques**: Algorithms adapting avatar aesthetics to user preferences 4. **Real-time Processing**: Optimized models for quick, on-device generation ### Challenges and Solutions - **Challenge**: Accurate facial detection across diverse demographics - **Solution**: Trained on diverse datasets with data augmentation techniques - **Challenge**: Balancing accuracy with artistic appeal - **Solution**: Developed scoring system balancing facial similarity with aesthetics - **Challenge**: Mobile device performance - **Solution**: Model compression and TensorFlow Lite optimization ### Results - 95% user satisfaction rate - 70% increase in avatar feature engagement - Avatar creation time reduced from minutes to seconds - 1M+ unique avatars processed in first month ## 2. Vernacular Sticker Keyboard: NLP and Federated Learning ### Project Overview Developed an AI-driven vernacular sticker keyboard that intelligently suggested stickers based on multilingual inputs, including Hinglish, Tamil English, and various language combinations. ### Technical Approach **Core Technologies:** - Python, TensorFlow, TensorFlow Lite - NLP techniques for language understanding - Federated learning for privacy-preserving updates **Key Features:** 1. **Multilingual Input Processing**: NLP models understanding mixed-language inputs 2. **Contextual Sticker Suggestion**: AI model suggesting relevant stickers based on text and context 3. **On-Device Personalization**: TensorFlow Lite models for on-device learning 4. **Federated Learning**: System for updating global models while maintaining privacy ### Challenges and Solutions - **Challenge**: Handling diverse linguistic combinations - **Solution**: Trained on vast multilingual corpus with advanced tokenization - **Challenge**: Real-time mobile performance - **Solution**: TensorFlow Lite optimization and efficient caching - **Challenge**: Balancing personalization with privacy - **Solution**: Federated learning allowing improvements without centralized data ### Results - 40% increase in sticker usage platform-wide - 60% improvement in suggestion relevance - Successfully handled 10+ language combinations - Privacy maintained through federated learning ## 3. Vibe Metaverse: Social Matchmaking ### Project Overview Developed a sophisticated AI-driven matchmaking system for Vibe, Hike's metaverse friendship network. The goal was creating meaningful connections by optimally selecting users for virtual rooms based on interests, interaction history, and social dynamics. ### Technical Approach **Core Technologies:** - Python, optimization solvers - BigQuery, Airflow, TensorFlow **Key Components:** 1. **User Profiling**: Comprehensive profiles based on interactions, preferences, and behavior 2. **Matchmaking Algorithm**: Advanced optimization algorithm for optimal room grouping 3. **Real-time Processing**: Real-time matchmaking decisions 4. **Performance Metrics**: KPIs measuring match success and user satisfaction ### Challenges and Solutions - **Challenge**: Balancing multiple matchmaking factors - **Solution**: Multi-objective optimization with weighted importance - **Challenge**: Ensuring diversity while maintaining relevance - **Solution**: Constraint-based approach mixing similar and diverse users - **Challenge**: Dynamic user preferences - **Solution**: Adaptive system continuously updating profiles ### Results - 50% increase in virtual room engagement - 40% improvement in social interaction satisfaction scores - 85% average room satisfaction rate - 60% reduction in inactive/abandoned rooms ## 4. Vibe Trust & Safety: Malicious Reporting Detection ### Project Overview Developed a sophisticated AI system to detect and mitigate malicious reporting within the Vibe metaverse, maintaining a safe, trustworthy environment. ### Technical Approach **Core Technologies:** - Python, modified PageRank algorithm - BigQuery, Airflow, TensorFlow **Key Components:** 1. **Trust Scoring System**: Modified PageRank assigning trust scores based on interactions and reporting history 2. **Behavioral Analysis**: Models analyzing user behavior patterns and identifying anomalies 3. **Report Classification**: ML model classifying reports by genuineness likelihood 4. **Real-time Processing**: Real-time analysis and decision-making ### Challenges and Solutions - **Challenge**: Distinguishing genuine from false reports - **Solution**: Multi-faceted approach combining trust scores, behavioral analysis, content evaluation - **Challenge**: Evolving malicious behavior - **Solution**: Adaptive system continually updating through machine learning - **Challenge**: Balancing swift action against false positives - **Solution**: Tiered response system with human oversight for high-stakes decisions ### Results - 75% reduction in false/malicious reports (first 3 months) - 40% improvement in user trust scores - 60% faster resolution of legitimate reports - 99.9% accuracy distinguishing genuine vs. malicious reports ## 5. Rush Gaming: AI-Driven Player Matchmaking ### Project Overview Developed an innovative AI-driven matchmaking system for Rush, Hike's real-money gaming network. The goal was creating fair, engaging, personalized gaming experiences by matching players based on skill levels, gaming behavior, and overall experience. ### Technical Approach **Core Technologies:** - Python, TensorFlow - BigQuery, Airflow - Custom ranking algorithms (ELO, TrueSkill-inspired) **Key Components:** 1. **Player Skill Evaluation**: Multi-faceted rating system considering game-specific skills and performance 2. **Behavioral Analysis**: Models analyzing play style, preferences, interaction patterns 3. **Real-time Matchmaking Engine**: High-performance system for instant decisions 4. **Fairness Assurance**: Algorithms ensuring balanced matches, detecting unfair advantages 5. **Adaptive Learning**: Continuous learning from match outcomes and feedback ### Challenges and Solutions - **Challenge**: Balancing match quality with wait times - **Solution**: Dynamic algorithm adjusting criteria based on queue times and player pool - **Challenge**: Ensuring fairness in diverse player ecosystem - **Solution**: Multi-dimensional ranking beyond win/loss ratios - **Challenge**: New player onboarding - **Solution**: Rapid assessment using initial games to gauge skill levels ### Results - 40% increase in player retention - 60% improvement in match quality ratings - 30% reduction in queue times - 50% reduction in negative gaming experiences ## Common Technical Infrastructure All five systems leveraged shared infrastructure: - **BigQuery**: Large-scale data storage and analysis - **Airflow**: Workflow orchestration and scheduling - **TensorFlow/PyTorch**: Model development and training - **Python**: Core development language ## Conclusion These five ML systems demonstrate the breadth and impact of AI in transforming mobile platform experiences. From computer vision and NLP to social matching, trust & safety, and gaming - each project solved unique challenges while serving millions of users. The success of these initiatives at Hike showcases how thoughtful application of machine learning can enhance user engagement, ensure platform integrity, and create more personalized, meaningful experiences at scale. --- *About the author: [Dipankar Sarkar](/about/) is a technology leader with deep expertise in machine learning and AI. As ML team leader at Hike, he pioneered multiple AI-driven features including matchmaking, computer vision, and NLP systems serving millions of users. [View all posts](/writings/) | [Get in touch](/contact/)* --- ## Building a Real-Time Data Ingestion and Analytics Framework for E-Commerce URL: https://www.dipankar.name/writings/building-realtime-data-ingestion-analytics-framework-ecommerce/ Date: 2024-01-10 Tags: data-engineering, real-time-analytics, aws, big-data, e-commerce Dipankar Sarkar built Nykaa's real-time data ingestion framework processing 5B+ daily events using AWS Lambda and Apache Flink, enabling data-driven personalization. As the Principal Engineering Consultant for a leading e-commerce platform in India, I spearheaded the development of a state-of-the-art real-time data ingestion and analytics framework. This project aimed to provide comprehensive, real-time insights into user behavior and system performance, surpassing the capabilities of traditional analytics tools like Adobe Analytics and Google Analytics. ## Project Overview Our objectives were to: 1. Develop a scalable, real-time data ingestion system capable of handling billions of events daily 2. Create a flexible analytics framework to process and analyze data in real-time 3. Provide actionable insights to various business units faster than ever before 4. Ensure data accuracy, security, and compliance with privacy regulations ## Technical Architecture ### Data Ingestion Layer - **AWS Lambda**: Used for serverless, event-driven data ingestion - **Amazon Kinesis**: For real-time data streaming - **Custom SDK**: Developed for client-side data collection across web and mobile platforms ### Data Processing and Storage - **Apache Flink**: For complex event processing and stream analytics - **Amazon S3**: As a data lake for storing raw and processed data - **Amazon Redshift**: For data warehousing and complex analytical queries ### Analytics and Visualization - **Custom Analytics Engine**: Built using Python and optimized for our specific needs - **Tableau and Custom Dashboards**: For data visualization and reporting ## Key Features 1. **Real-Time Event Processing**: Capability to ingest and process billions of events daily with sub-second latency 2. **Customizable Event Tracking**: Flexible system allowing easy addition of new event types and attributes 3. **User Journey Analysis**: Advanced tools for tracking and analyzing complete user journeys across multiple sessions and devices 4. **Predictive Analytics**: Machine learning models for predicting user behavior and product trends 5. **A/B Testing Framework**: Integrated system for running and analyzing A/B tests in real-time 6. **Anomaly Detection**: Automated systems for detecting unusual patterns in user behavior or system performance ## Implementation Challenges and Solutions 1. **Challenge**: Handling massive data volume and velocity **Solution**: Implemented a distributed, scalable architecture using AWS services and optimized data partitioning strategies 2. **Challenge**: Ensuring data consistency and accuracy **Solution**: Developed robust data validation and reconciliation processes, with automated alerts for data discrepancies 3. **Challenge**: Balancing real-time processing with historical analysis **Solution**: Created a lambda architecture, combining stream processing for real-time insights with batch processing for in-depth historical analysis 4. **Challenge**: Compliance with data privacy regulations **Solution**: Implemented data anonymization techniques and strict access controls, ensuring compliance with GDPR and local data protection laws ## Development Process 1. **Requirements Gathering**: Conducted extensive interviews with various business units to understand their analytics needs 2. **Proof of Concept**: Developed a small-scale prototype to validate the architecture and core functionalities 3. **Incremental Development**: Adopted an agile approach, releasing features incrementally and gathering feedback 4. **Performance Optimization**: Conducted extensive load testing and optimization to handle peak traffic scenarios 5. **Training and Documentation**: Created comprehensive documentation and conducted training sessions for data analysts and business users ## Results and Impact 1. **Data Processing Capability**: - Successfully ingested and processed over 5 billion events daily - Reduced data latency from hours to seconds 2. **Cost Efficiency**: - 40% reduction in data analytics costs compared to previous third-party solutions 3. **Business Impact**: - 25% improvement in conversion rates through real-time personalization - 30% increase in customer retention through better-targeted campaigns 4. **Operational Efficiency**: - 50% reduction in time spent on data preparation and analysis by data science teams ## Future Enhancements 1. Integrating advanced AI/ML models for deeper predictive analytics 2. Expanding the system to include more IoT data sources 3. Developing a self-service analytics platform for non-technical users ## Related Reading More e-commerce platform work at Nykaa: - [Building Scalable E-Commerce Infrastructure](/writings/building-scalable-ecommerce-infrastructure-platform-migration/) - Platform migration with in-memory cart service and API gateway - [Real-Time Personalized Feed for E-Commerce](/writings/innovating-user-engagement-realtime-personalized-feed-ecommerce/) - TikTok-inspired discovery and engagement - [Integrated Ad Platform and Social Commerce](/writings/revolutionizing-ecommerce-integrated-ad-platform-social-commerce/) - Revenue-driving advertising solutions ## Conclusion The development of our real-time data ingestion and analytics framework marked a significant milestone in our e-commerce platform's data capabilities. By moving beyond traditional analytics tools and building a custom solution tailored to our specific needs, we've gained unprecedented insights into user behavior and system performance. This project not only enhanced our ability to make data-driven decisions but also positioned us at the forefront of e-commerce analytics. The real-time nature of our new system allows for immediate responses to market trends and user behaviors, giving us a competitive edge in the fast-paced e-commerce landscape. As we continue to evolve and expand this system, it remains a cornerstone of our data strategy, driving innovation and growth across all aspects of our e-commerce operations. The success of this project demonstrates the immense value of investing in custom, cutting-edge data solutions in today's data-driven business environment. --- *About the author: [Dipankar Sarkar](/about/) is a technology leader specializing in data engineering and real-time analytics. As Principal Engineering Consultant at Nykaa, he architected scalable data platforms processing billions of events. [View all posts](/writings/) | [Get in touch](/contact/)* --- ## RoboGPT: Natural Language Robot Programming and Industry Transformation URL: https://www.dipankar.name/writings/robogpt-natural-language-robot-programming-industry-transformation/ Date: 2023-10-15 Tags: llm, robotics, nlp, industrial-automation, machine-learning, ai Dipankar Sarkar developed RoboGPT at Orangewood Labs - an LLM-powered interface enabling natural language robot programming, transforming industrial automation across manufacturing and healthcare. As the Head of AI & Platform at Orangewood Labs, I led the development of RoboGPT, our groundbreaking solution that's transforming the robotics industry. By leveraging the power of Large Language Models (LLMs), we created a voice and text-enabled interface for high-level planning with collaborative robots (cobots), eliminating the need for manual programming and accelerating low-level cognition. ## The Challenge: Bridging the Gap Between Humans and Robots Traditionally, programming robots has been a complex task requiring specialized knowledge and skills. This complexity has been a significant barrier to the widespread adoption of robotics in various industries. Our goal with RoboGPT was to make robot programming as intuitive as having a conversation, allowing even non-technical users to interact with and control robots effectively. ## RoboGPT: Natural Language Programming for Robots RoboGPT represents a paradigm shift in how we interact with robots. Here's how it works: 1. **Natural Language Input**: Users can give instructions to robots using voice or text, just as they would communicate with a human colleague. 2. **LLM-Powered Understanding**: Our advanced LLM processes the natural language input, understanding context, intent, and nuances. 3. **High-Level Planning**: RoboGPT translates the user's instructions into high-level plans for the robot to execute. 4. **Low-Level Execution**: These high-level plans are then broken down into specific actions that the robot can perform. 5. **Feedback Loop**: The robot provides feedback on its actions, which RoboGPT translates back into natural language for the user. ## Key Advantages of RoboGPT 1. **Accessibility**: Non-programmers can now effectively work with robots, broadening the potential user base. 2. **Flexibility**: Quickly adapt robot behavior to new tasks without extensive reprogramming. 3. **Efficiency**: Reduce the time and cost associated with robot deployment and task switching. 4. **Enhanced Collaboration**: Improve human-robot interaction in collaborative workspaces. 5. **Continuous Learning**: The system can learn from interactions, continuously improving its understanding and capabilities. ## Industry Impact: Real-World Success Stories ### Manufacturing: Agile Production Lines In the manufacturing sector, RoboGPT has enabled unprecedented flexibility: - **Rapid Retooling**: A major automotive manufacturer reported a 70% reduction in production line reconfiguration time, allowing for quick adaptation to new models or customizations. - **Skill Democratization**: Small and medium-sized enterprises have seen a 50% increase in the adoption of robotic systems, as RoboGPT lowers the barrier to entry for non-technical staff. - **Easy Reconfiguration**: Easily reconfigure assembly line robots for different products. Combined with our [ML-driven precision systems for industrial robotics](/writings/autoinspect-autospray-ml-driven-precision-industrial-robotics/), we're transforming quality control and automation. ### Healthcare: Precision and Accessibility RoboGPT is making waves in healthcare robotics: - **Surgical Assistance**: Surgeons can now give voice commands to robotic surgical assistants, enhancing precision and reducing fatigue during long procedures. Medical staff can operate specialized robotic equipment with ease. - **Rehabilitation Robotics**: Physical therapists are using RoboGPT to easily customize rehabilitation robots for individual patient needs, leading to a 40% improvement in patient outcomes. ### Agriculture: Smart Farming Revolution The agricultural sector has seen significant advancements: - **Adaptive Harvesting**: Farmers are using RoboGPT to quickly reprogram harvesting robots for different crops, increasing efficiency by 35%. Farming robots can easily adapt to different crops and conditions. - **Precision Agriculture**: Drones and ground robots are being easily instructed to perform targeted tasks like pest control and soil analysis, reducing chemical use by 50%. ### Research and Development: Accelerating Innovation RoboGPT is proving invaluable in research settings: - **Lab Automation**: Scientists report a 60% increase in experiment throughput, as they can rapidly instruct lab robots to perform complex procedures. Researchers can quickly set up and modify experimental robotic systems. - **Space Exploration**: NASA is exploring RoboGPT for more flexible control of rovers on distant planets, potentially revolutionizing space exploration. ## Enhancing Human-Robot Collaboration RoboGPT isn't just about making robots easier to program; it's fundamentally changing how humans and robots interact: 1. **Natural Communication**: Workers report feeling more comfortable and confident when working alongside robots they can communicate with naturally. 2. **Continuous Learning**: RoboGPT-enabled robots can learn from human instructions, continuously improving their capabilities. 3. **Contextual Understanding**: The system's ability to understand context allows for more nuanced and efficient human-robot teamwork. 4. **Safety Enhancements**: Natural language interactions enable quicker and more intuitive safety commands, improving workplace safety. ## Challenges and Solutions As with any transformative technology, RoboGPT has faced challenges: 1. **Language Diversity**: We've expanded language support to over 50 languages, ensuring global accessibility. 2. **Privacy Concerns**: Implemented advanced encryption and local processing options to address data privacy issues. 3. **Integration with Legacy Systems**: Developed adapters and middleware solutions to ensure compatibility with existing industrial equipment. ## The Road Ahead: Future Directions for RoboGPT As we continue to refine and expand RoboGPT, we're exploring several exciting avenues: 1. **Multimodal Interaction**: Integrating visual inputs to allow robots to understand and respond to gestures and environmental cues. 2. **Enhanced Contextual Understanding**: Improving the system's ability to understand and maintain context over extended interactions. 3. **Task Generalization**: Developing the ability for robots to apply learned skills to novel situations. 4. **Inter-Robot Communication**: Enabling robots to share knowledge and coordinate tasks using natural language. Extending RoboGPT to facilitate communication and coordination among multiple robots for large-scale operations (swarm intelligence). 5. **Edge Computing Integration**: Leveraging our [EdgeML robotics platform](/writings/edgeml-future-robotics-next-generation-sdk-platform/) to enable on-device processing for faster, more responsive robot control. 6. **Emotional Intelligence**: Developing capabilities for robots to recognize and respond appropriately to human emotions, enhancing collaboration in sensitive environments like healthcare. 7. **Augmented Reality Integration**: Combining RoboGPT with AR to provide visual feedback and instructions, creating a more immersive human-robot collaboration experience. 8. **Predictive Assistance**: Enhancing RoboGPT with predictive models to anticipate human needs and proactively offer assistance. ## Related Reading If you enjoyed this article, you might also be interested in: - [AutoInspect and AutoSpray: ML-Driven Precision in Industrial Robotics](/writings/autoinspect-autospray-ml-driven-precision-industrial-robotics/) - How we're applying machine learning to industrial quality control and spray painting - [EdgeML and the Future of Robotics](/writings/edgeml-future-robotics-next-generation-sdk-platform/) - Building the next-generation SDK and platform for intelligent robotic systems - [Machine Learning at Hike](/writings/machine-learning-hike-ai-systems/) - Leading ML team developing five AI systems serving millions ## Conclusion: A New Era of Human-Robot Collaboration RoboGPT represents more than just a technological advancement; it's a bridge between human creativity and robotic precision. By breaking down the communication barriers between humans and robots, we're not only enhancing productivity and efficiency but also opening up new possibilities for innovation across industries. By making robots more accessible and easier to work with, we're opening up new possibilities for innovation and productivity across countless fields. The future of work is collaborative, intuitive, and intelligent – and RoboGPT is leading the way. As we move forward, we're excited to see how RoboGPT will continue to evolve and shape the future of robotics. The era of intuitive, natural language-driven robotics is here, and at Orangewood Labs, we're proud to be at the forefront of this revolution. --- *About the author: [Dipankar Sarkar](/about/) is a technology leader with 18+ years of experience in AI, robotics, and distributed systems. Previously Head of AI & Platform at Orangewood Labs, he led development of RoboGPT and other groundbreaking robotics solutions. [View all posts](/writings/) | [Get in touch](/contact/)* --- ## Innovating User Engagement: Developing a Real-Time Personalized Feed for E-Commerce URL: https://www.dipankar.name/writings/innovating-user-engagement-realtime-personalized-feed-ecommerce/ Date: 2023-08-05 Tags: recommendation-systems, machine-learning, personalization, e-commerce, real-time-systems Dipankar Sarkar built a TikTok-inspired real-time personalized feed at Nykaa using AI/ML, achieving 200% increase in daily active users and 150% more engagement. As the Principal Engineering Consultant for a leading e-commerce platform in India, I led the development of a groundbreaking feature: a real-time personalized feed that revolutionized how users discover and engage with content within our application. This TikTok-inspired feature, tailored for e-commerce, significantly enhanced user engagement and time spent on the platform. ## Project Overview Our goal was to create a dynamic, engaging feed that would: 1. Provide personalized, relevant content to each user in real-time 2. Increase user engagement and time spent on the app 3. Drive product discovery and sales 4. Leverage user-generated content alongside curated brand content ## Technical Approach ### Key Components 1. **Content Aggregation System**: Collects and processes various types of content (user-generated, brand-created, product information) 2. **Real-Time Personalization Engine**: Utilizes AI/ML to deliver personalized content to each user 3. **Tag-Based Content Classification**: Implements a sophisticated tagging system for efficient content categorization and retrieval 4. **High-Performance Content Delivery**: Ensures smooth, buffer-free content streaming ### Technology Stack - **Backend**: Python with FastAPI for high-performance API endpoints - **Machine Learning**: TensorFlow and PyTorch for recommendation models - **Real-Time Processing**: Apache Kafka and Flink for stream processing - **Database**: MongoDB for content metadata, Redis for caching - **Content Delivery**: AWS CloudFront and Elastic Transcoder for video processing and delivery ## Key Features 1. **Personalized Content Ranking**: Developed an algorithm that ranks content based on user preferences, behavior, and real-time engagement metrics 2. **Interactive Elements**: Implemented features like likes, comments, and shares to increase user engagement 3. **Seamless Product Integration**: Created a system to seamlessly integrate product information and purchase options within the content feed 4. **Content Creator Tools**: Developed in-app tools for users and brands to create and upload engaging content directly 5. **A/B Testing Framework**: Implemented a robust A/B testing system to continuously optimize the feed algorithm ## Challenges and Solutions 1. **Challenge**: Achieving real-time personalization at scale **Solution**: Implemented a hybrid approach combining pre-computed recommendations with real-time adjustments 2. **Challenge**: Balancing diverse content types (user-generated, promotional, educational) **Solution**: Developed a content mix algorithm that optimizes for user engagement while meeting business objectives 3. **Challenge**: Ensuring content relevance and quality **Solution**: Implemented an AI-driven content moderation system and user reputation algorithm ## Implementation Process 1. **Data Collection and Analysis**: Gathered and analyzed user behavior data to inform the personalization algorithm 2. **Prototype Development**: Created a MVP to test core functionalities and gather user feedback 3. **Scalability Testing**: Conducted extensive load testing to ensure the system could handle millions of concurrent users 4. **Gradual Rollout**: Implemented the feature in phases, starting with a small user group and gradually expanding 5. **Continuous Optimization**: Established a process for ongoing algorithm refinement based on user engagement metrics ## Results and Impact 1. **User Engagement**: - 200% increase in daily active users - 150% increase in average time spent on the app 2. **Content Creation**: - 500% increase in user-generated content within the first three months 3. **Sales Performance**: - 30% increase in click-through rates to product pages - 25% boost in conversion rates for products featured in the feed 4. **Technical Performance**: - Achieved sub-100ms latency for content recommendations - Scaled to handle over 5000+ concurrent users ## Conclusion The development of our real-time personalized feed marked a significant leap forward in e-commerce user engagement. By blending the addictive nature of short-form video content with personalized product recommendations, we created a unique and compelling user experience that drove both engagement and sales. ## Related Reading More e-commerce innovation work: - [Integrated Ad Platform and Social Commerce](/writings/revolutionizing-ecommerce-integrated-ad-platform-social-commerce/) - Influencer networks and affiliate marketing - [Real-Time Data Ingestion Framework](/writings/building-realtime-data-ingestion-analytics-framework-ecommerce/) - Analytics infrastructure powering personalization - [Building Scalable E-Commerce Infrastructure](/writings/building-scalable-ecommerce-infrastructure-platform-migration/) - Platform migration and high-performance services This project showcased the power of combining cutting-edge technologies in AI, real-time data processing, and content delivery to create a feature that resonates with modern users' preferences for dynamic, personalized content. As we continue to refine and expand this feature, it remains a cornerstone of our strategy to keep users engaged, drive product discovery, and stay at the forefront of e-commerce innovation. The success of this project has not only transformed our platform but also set new standards for user engagement in the e-commerce industry. --- *About the author: [Dipankar Sarkar](/about/) is a technology leader specializing in AI/ML and personalization. As Principal Engineering Consultant at Nykaa, he developed innovative engagement features driving significant user growth. [View all posts](/writings/) | [Get in touch](/contact/)* --- ## Revolutionizing E-Commerce: Building an Integrated Ad Platform and Social Commerce Solution URL: https://www.dipankar.name/writings/revolutionizing-ecommerce-integrated-ad-platform-social-commerce/ Date: 2023-02-18 Tags: e-commerce, advertising-platform, machine-learning, social-commerce, marketplace Dipankar Sarkar built Nykaa's integrated ad platform and social commerce solution, achieving 200% revenue increase and onboarding 1000+ influencers. As the Principal Engineering Consultant for a major e-commerce player in India, I led the development of two groundbreaking platforms that significantly boosted our revenue streams and user engagement: an advanced Ad Platform and an innovative Social Commerce solution. These projects not only enhanced our digital marketing capabilities but also positioned us at the forefront of e-commerce innovation. ## Project Overview Our objectives were to: 1. Create a robust Ad Platform to monetize our high traffic and provide value to brand partners 2. Develop a Social Commerce platform to leverage user-generated content and increase engagement 3. Implement a flexible architecture to support affiliate and influencer networks ## Ad Platform Development ### Key Features 1. **Targeted Ad Placements**: Developed an algorithm for contextual and user-preference based ad targeting 2. **Real-Time Bidding**: Implemented a real-time bidding system for ad inventory 3. **Performance Analytics**: Created comprehensive dashboards for advertisers to track campaign performance 4. **Multi-Format Ads**: Supported various ad formats including banner ads, product listings, and video ads ### Technical Implementation - Used Python and Django for the backend services - Implemented Elasticsearch for fast, real-time ad serving - Utilized Redis for caching and real-time data processing - Deployed on AWS for scalability and reliability ### Challenges and Solutions 1. **Challenge**: Balancing ad relevance with user experience **Solution**: Developed a machine learning model to optimize ad placements based on user engagement metrics 2. **Challenge**: Handling high-volume, real-time bidding **Solution**: Implemented a distributed system using Apache Kafka for processing bid requests and responses ## Social Commerce Platform ### Key Features 1. **User-Generated Content**: Developed a platform for users to create and share product-related content 2. **Shoppable Posts**: Implemented functionality to make user posts shoppable with direct product links 3. **Influencer Dashboard**: Created tools for influencers to track their performance and earnings 4. **Personalized Feed**: Developed an AI-driven personalized feed algorithm ### Technical Implementation - Utilized a microservices architecture with Python-based services - Used Collaborative filtering for AI-driven content recommendation - Leveraged AWS S3 and CloudFront for content delivery ### Challenges and Solutions 1. **Challenge**: Ensuring content quality and relevance **Solution**: Implemented an AI-based content moderation system and user reputation algorithm 2. **Challenge**: Real-time personalization at scale **Solution**: Developed a hybrid recommendation system combining collaborative filtering and content-based approaches ## Affiliate and Influencer Network Integration ### Key Features 1. **Flexible Commission Structure**: Implemented a configurable commission system for different product categories and influencer tiers 2. **Attribution Tracking**: Developed a robust system to track and attribute sales to specific affiliates or influencers 3. **Payment Integration**: Automated payment processing for affiliates and influencers ### Technical Implementation - Used blockchain technology for transparent and immutable transaction recording - Implemented smart contracts for automated commission calculations and payments - Developed RESTful APIs for easy integration with external influencer platforms ## Results and Impact 1. **Ad Platform Performance**: - 200% increase in advertising revenue within the first year - 30% improvement in click-through rates for targeted ads 2. **Social Commerce Success**: - 50% increase in user-generated content - 40% higher conversion rate for products featured in user-generated content 3. **Affiliate and Influencer Network**: - Onboarded 1000+ influencers within six months - 25% of total sales attributed to affiliate and influencer marketing ## Conclusion The development of our integrated Ad Platform and Social Commerce solution marked a significant milestone in our e-commerce journey. By leveraging cutting-edge technologies and innovative approaches, we not only created new revenue streams but also enhanced user engagement and brand loyalty. ## Related Reading More revenue-driving platform work: - [Real-Time Personalized Feed](/writings/innovating-user-engagement-realtime-personalized-feed-ecommerce/) - Content discovery and user engagement - [Real-Time Data Ingestion Framework](/writings/building-realtime-data-ingestion-analytics-framework-ecommerce/) - Analytics powering ad targeting - [Building Scalable E-Commerce Infrastructure](/writings/building-scalable-ecommerce-infrastructure-platform-migration/) - Platform migration enabling rapid innovation These platforms demonstrated the power of combining traditional e-commerce with social media dynamics and targeted advertising. The flexible architecture we implemented allowed for seamless integration of affiliate and influencer networks, further amplifying our reach and sales potential. As we continue to evolve in the dynamic e-commerce landscape, these innovations remain central to our strategy, enabling us to stay ahead of market trends and provide unparalleled value to both our users and brand partners. --- *About the author: [Dipankar Sarkar](/about/) is a technology leader specializing in digital marketing platforms and social commerce. As Principal Engineering Consultant at Nykaa, he built revenue-driving ad and influencer platforms. [View all posts](/writings/) | [Get in touch](/contact/)* --- ## Pioneering India's First True E-commerce Marketplace at Tradus URL: https://www.dipankar.name/writings/pioneering-indias-first-ecommerce-marketplace-tradus/ Date: 2022-03-15 Tags: e-commerce, marketplace, api-design, team-leadership, startup-scaling How we built India's first true multi-seller e-commerce marketplace at Tradus. Pioneering public APIs, 15-engineer team, and the technical decisions that drove 300% listing growth. In the early 2010s, as India's e-commerce landscape was just beginning to take shape, I had the opportunity to lead a transformative project at Tradus. As the Senior Engineering Manager, I was tasked with developing India's first true e-commerce marketplace, a challenge that would push the boundaries of what was possible in the country's nascent online retail space. ## The Tradus Vision Tradus (https://tradus.com) aimed to revolutionize online shopping in India by creating a platform where multiple sellers could list their products, competing on price and service quality. This was a novel concept in India at the time, where most e-commerce sites operated on an inventory-based model. ## Team Leadership and Project Scope Managing a team of 15 engineers, our mission was clear but challenging: 1. Transform the existing Tradus platform into a full-fledged marketplace 2. Develop India's first public e-commerce APIs for crawlers and aggregators 3. Enhance the user interface with sophisticated widgets and JavaScript libraries 4. Optimize the platform's performance to handle increased traffic and transactions ## Building the Marketplace ### Marketplace Architecture We started by redesigning the core architecture to support multiple sellers, implementing: - A scalable seller onboarding system - Inventory management tools for sellers - A robust order routing and fulfillment system - A fair and transparent seller rating mechanism ### Pioneering E-commerce APIs One of our most significant achievements was designing and implementing India's first public e-commerce APIs. This involved: - Creating RESTful APIs for product listings, pricing, and availability - Implementing OAuth for secure API access - Developing comprehensive API documentation - Building a developer portal for API users These APIs were a game-changer, allowing crawlers and aggregators to access Tradus data, significantly increasing our product visibility across the web. ### Revamping the User Interface To enhance the user experience, we: - Implemented a new, responsive design using HTML5 and CSS3 - Developed custom JavaScript widgets for dynamic content loading - Integrated advanced search and filtering capabilities - Created an intuitive category navigation system ### Performance Optimization As traffic grew, maintaining performance became crucial. We focused on: - Implementing aggressive caching strategies - Optimizing database queries - Setting up content delivery networks (CDNs) for static assets - Implementing asynchronous processing for non-critical tasks ## Challenges and Solutions ### Challenge: Legacy System Integration Integrating new marketplace features with the existing legacy system posed significant challenges. **Solution**: We adopted a microservices architecture, gradually migrating functionalities from the monolithic system to new, scalable services. ### Challenge: Seller Adoption Convincing sellers to list on a new marketplace platform was initially difficult. **Solution**: We developed easy-to-use seller tools and provided dedicated support to help sellers transition to the platform. ### Challenge: Ensuring Fair Competition Balancing the interests of both large and small sellers on the platform was crucial for long-term success. **Solution**: We implemented a sophisticated ranking algorithm that considered factors beyond just price, including seller ratings, shipping speed, and customer service quality. ## Impact and Legacy The launch of the Tradus marketplace was a landmark moment in Indian e-commerce: - We became the first true multi-seller marketplace in India, predating competitors like ShopClues. - The platform saw a 300% increase in product listings within the first six months. - Our public APIs were adopted by major price comparison and product discovery platforms, significantly increasing our reach. - The enhanced user interface and performance optimizations led to a 50% increase in conversion rates. ## Related Reading More e-commerce platform work: - [Building Scalable E-Commerce Infrastructure](/writings/building-scalable-ecommerce-infrastructure-platform-migration/) - Modern architecture with in-memory cart service and API gateway - [Building Analytical Systems at Tyroo](/writings/building-analytical-systems-core-java-tyroo-adtech/) - AdTech analytics and big data ## Conclusion Leading the development of India's first true e-commerce marketplace at Tradus was a defining experience in my career. It showcased the power of innovative thinking, strong team leadership, and cutting-edge technology in transforming an industry. The lessons learned and the technologies we pioneered continue to influence e-commerce development in India to this day. As e-commerce continues to evolve, the principles of open APIs, performance optimization, and seller empowerment that we championed at Tradus remain more relevant than ever. This project not only revolutionized online shopping in India but also set new standards for what was possible in the realm of e-commerce technology. --- *About the author: [Dipankar Sarkar](/about/) is a technology leader with 18+ years building scalable platforms. As Senior Engineering Manager at Tradus, he pioneered India's first true e-commerce marketplace. [View all posts](/writings/) | [Get in touch](/contact/)* --- ## Building Scalable E-Commerce Infrastructure: Platform Migration and High-Performance Services URL: https://www.dipankar.name/writings/building-scalable-ecommerce-infrastructure-platform-migration/ Date: 2021-09-15 Tags: e-commerce, platform-architecture, microservices, python, scalability Dipankar Sarkar led Nykaa's platform migration from Magento to custom Python, implementing in-memory cart service and Kong API gateway for 500% traffic growth. As a Principal Engineering Consultant for a leading e-commerce company in India, I spearheaded a comprehensive infrastructure transformation that modernized our technology stack and enabled unprecedented scalability. This multi-faceted project included migrating from Magento to a custom Python-based platform and implementing critical high-performance services for cart management and API routing. ## The Challenge Our rapidly growing e-commerce platform was facing significant limitations: 1. Scalability issues during high-traffic events 2. Cart abandonment due to slow response times during peak periods 3. Limited flexibility for implementing custom features 4. Performance bottlenecks affecting user experience 5. Difficulty in managing and scaling our growing number of microservices 6. High operational costs due to licensing and hosting requirements 7. Need for better traffic management and security at the API level ## Platform Migration: From Magento to Custom Python We embarked on a comprehensive migration plan, choosing Python as the core language for our new platform. Key aspects of our approach included: ### Migration Strategy 1. **Gradual Migration**: We adopted a phased approach, gradually moving components from Magento to our new Python-based system. 2. **Microservices Architecture**: We broke down the monolithic Magento structure into microservices, enhancing modularity and scalability. 3. **Python Ecosystem**: We leveraged Python's rich ecosystem, utilizing frameworks like Django and Flask for different components of our system. 4. **API-First Design**: We implemented an API-first approach, facilitating easier integration with mobile apps and third-party services. 5. **Cloud-Native Architecture**: The new platform was designed to be cloud-native, taking full advantage of scalable cloud services. ### Implementation Highlights 1. **Core Services in Python**: We rewrote critical services like product catalog, and user authentication in Python and Java. 2. **Data Migration**: Developed robust ETL processes to migrate data from Magento to our new database structure. 3. **Performance Optimization**: Implemented caching strategies and optimized database queries to enhance overall system performance. 4. **DevOps Integration**: Set up CI/CD pipelines for automated testing and deployment of our Python-based services. 5. **Monitoring and Logging**: Implemented comprehensive monitoring and logging solutions for better system observability. ### Migration Challenges and Solutions 1. **Challenge**: Ensuring business continuity during migration. **Solution**: Implemented a strangler pattern, gradually replacing Magento components while maintaining seamless operation. 2. **Challenge**: Knowledge transition from Magento to Python ecosystem. **Solution**: Conducted intensive training sessions and pair programming to upskill the development team. 3. **Challenge**: Maintaining data integrity during migration. **Solution**: Developed rigorous data validation and reconciliation processes to ensure data accuracy. ## In-Memory SQL-Based Cart Service A critical component of our new infrastructure was a high-performance, scalable solution for managing user shopping carts. ### Design Principles 1. **Speed**: Utilize in-memory processing for ultra-fast read/write operations. 2. **Scalability**: Design for horizontal scalability to handle traffic spikes. 3. **Reliability**: Implement data persistence and recovery mechanisms. ### Implementation Details 1. **Technology Stack**: - Redis as the primary in-memory data store - SQLite for data persistence - Python for service logic 2. **Key Features**: - Real-time cart updates and synchronization - Session management for guest and logged-in users - Intelligent caching of product information 3. **Scalability Measures**: - Implemented sharding based on user IDs - Designed for easy replication and cluster management 4. **Data Consistency**: - Implemented a write-through caching strategy - Periodic snapshots for data persistence ### Cart Service Challenges and Solutions 1. **Challenge**: Ensuring data consistency in the distributed cart service. **Solution**: Implemented a distributed locking mechanism and eventual consistency model. 2. **Challenge**: Optimizing performance under high load. **Solution**: Implemented aggressive caching strategies and conducted extensive load testing to fine-tune our configurations. ## Kong-Based API Gateway We implemented a centralized gateway to manage, secure, and optimize API traffic across our microservices. ### Design Principles 1. **Centralized Management**: Single point of entry for all API requests. 2. **Security**: Robust authentication and authorization mechanisms. 3. **Performance**: Efficient routing and load balancing. ### Implementation Details 1. **Technology Stack**: - Kong API Gateway - Cassandra for storing Kong's configuration data - Lua for custom plugins 2. **Key Features**: - JWT authentication and rate limiting - Request/response transformation - Advanced load balancing - Analytics and monitoring integration 3. **Custom Plugins**: - Developed custom plugins for business-specific requirements - Implemented a caching layer for frequently accessed data 4. **Scalability Measures**: - Deployed Kong in a clustered configuration - Implemented blue-green deployment for zero-downtime updates ### API Gateway Challenges and Solutions 1. **Challenge**: Managing the complexity of routing logic in the API gateway. **Solution**: Developed a declarative configuration system for easy management of routing rules. 2. **Challenge**: Optimizing performance under high load. **Solution**: Implemented aggressive caching strategies and conducted extensive load testing to fine-tune our configurations. ## Results and Impact ### Platform Migration Results 1. **Improved Scalability**: The new platform easily handled a 500% increase in traffic during peak sales events. 2. **Enhanced Performance**: Page load times improved by 60%, significantly enhancing user experience. 3. **Cost Reduction**: Operational costs decreased by 40% due to optimized cloud resource utilization. 4. **Increased Development Velocity**: New feature development time reduced by 50%, thanks to the flexibility of our custom platform. 5. **Better Analytics**: Improved data collection and analysis capabilities, providing deeper insights into user behavior and business performance. ### Cart Service Performance 1. 99.99% uptime even during peak sale events 2. 95% reduction in cart abandonment due to technical issues 3. Ability to handle 100,000+ concurrent cart operations ### API Gateway Improvements 1. 50% reduction in API latency 2. Enhanced security with 99.9% blocking of malicious requests 3. Simplified microservices management and deployment ## Related Reading More e-commerce and platform work: - [Real-Time Data Ingestion Framework](/writings/building-realtime-data-ingestion-analytics-framework-ecommerce/) - Event-driven analytics at scale - [Real-Time Personalized Feed](/writings/innovating-user-engagement-realtime-personalized-feed-ecommerce/) - TikTok-inspired content discovery - [India's First E-commerce Marketplace at Tradus](/writings/pioneering-indias-first-ecommerce-marketplace-tradus/) - Early e-commerce platform development - [Integrated Ad Platform](/writings/revolutionizing-ecommerce-integrated-ad-platform-social-commerce/) - Revenue-driving advertising infrastructure ## Conclusion The comprehensive transformation of our e-commerce infrastructure - from Magento migration to implementing high-performance cart service and API gateway - marked a pivotal moment in our company's technological evolution. It not only solved immediate scalability and performance issues but also positioned us for future growth and innovation. The implementation of our in-memory cart service and Kong-based API gateway, combined with the flexibility of our Python-based platform, created a highly responsive, scalable, and secure e-commerce infrastructure. These innovations laid a robust foundation for handling the complexities of modern, high-traffic e-commerce platforms. The success of this project demonstrated the power of custom solutions in addressing unique business needs in the fast-paced world of e-commerce. As we continue to evolve our platform, the flexibility and scalability afforded by our Python-based architecture and high-performance services remain key drivers of our technological strategy, enabling us to deliver exceptional shopping experiences to millions of users, even during the most demanding peak periods. --- *About the author: [Dipankar Sarkar](/about/) is a technology leader specializing in platform architecture and scalable systems. As Principal Engineering Consultant at Nykaa, he led major platform modernization initiatives and architected high-performance infrastructure handling millions of transactions. [View all posts](/writings/) | [Get in touch](/contact/)* --- ## Building Analytical Systems in Core Java at Tyroo: Revolutionizing AdTech in India URL: https://www.dipankar.name/writings/building-analytical-systems-core-java-tyroo-adtech/ Date: 2021-08-20 Tags: big-data, analytics, java, advertising-platform, real-time-systems Dipankar Sarkar built analytical systems at Tyroo (India's largest AdTech company) processing 10B+ daily ad impressions using Core Java for real-time insights. In the early 2010s, as digital advertising was gaining momentum in India, I had the opportunity to work at Tyroo, the country's largest adtech company at the time. As a Software Engineer, my role was pivotal in developing analytical systems that would shape the future of data-driven advertising in the region. ## The Tyroo Vision Tyroo aimed to provide advertisers and publishers with deep insights into campaign performance, user behavior, and ROI. Our goal was to build robust, scalable analytical systems that could process vast amounts of advertising data in real-time. ## Technical Challenges and Solutions ### Handling Big Data The sheer volume of advertising data was our primary challenge. We needed to process billions of ad impressions, clicks, and conversions daily. **Solution**: We leveraged Core Java's efficiency to build a distributed processing system. Using technologies like Apache Hadoop for distributed storage and processing, we created a scalable infrastructure capable of handling terabytes of data. ### Real-Time Analytics Advertisers needed up-to-the-minute insights to optimize their campaigns effectively. **Solution**: We developed a real-time analytics engine using Java NIO (New I/O) for non-blocking I/O operations. This allowed us to process incoming data streams efficiently, providing near-real-time updates to our analytics dashboards. ### Complex Query Processing Advertisers often needed to run complex, multi-dimensional queries across vast datasets. **Solution**: We implemented a custom query engine using Java, optimized for the specific structure of our advertising data. This engine utilized advanced indexing techniques and in-memory caching to deliver rapid query results. ### Scalability and Performance As Tyroo's client base grew, our systems needed to scale seamlessly. **Solution**: We designed our applications with horizontal scalability in mind. Using Java's concurrency utilities, we built systems that could efficiently utilize multi-core processors and could be easily deployed across multiple servers. ## Key Features Developed 1. **Real-Time Dashboard**: A Java Swing-based desktop application for real-time monitoring of ad campaign performance. 2. **Predictive Analytics**: Implementing machine learning algorithms in Java to predict campaign performance and suggest optimizations. 3. **Fraud Detection System**: A sophisticated system using statistical analysis to identify and flag potentially fraudulent ad activities. 4. **Custom Reporting Engine**: A flexible reporting system allowing advertisers to generate custom reports with drag-and-drop simplicity. ## Challenges Overcome ### Challenge: Data Accuracy Ensuring the accuracy of data across millions of transactions was crucial for maintaining client trust. **Solution**: We implemented a multi-layer validation system, using Java's strong typing and custom validation algorithms to ensure data integrity at every step of the processing pipeline. ### Challenge: System Latency As data volumes grew, maintaining low latency became increasingly difficult. **Solution**: We optimized our Java code rigorously, utilizing profiling tools to identify and eliminate bottlenecks. We also implemented a caching layer using Ehcache to reduce database load for frequently accessed data. ### Challenge: Integration with Multiple Ad Networks Tyroo needed to integrate with various ad networks, each with its own data format and APIs. **Solution**: We developed a flexible adapter system in Java, allowing easy integration of new ad networks with minimal code changes. This system used Java interfaces and abstract classes to create a standardized way of handling data from different sources. ## Impact and Legacy Our work at Tyroo had a significant impact on the Indian adtech landscape: - Processed over 10 billion ad impressions daily, providing insights to thousands of advertisers. - Reduced campaign optimization time by 60%, allowing advertisers to respond more quickly to market changes. - Improved fraud detection rates by 40%, significantly increasing the value of ad spends for clients. - Set new industry standards for data processing speed and accuracy in the Indian adtech sector. ## Related Reading More early career platform work: - [India's First E-commerce Marketplace at Tradus](/writings/pioneering-indias-first-ecommerce-marketplace-tradus/) - Building marketplace platforms and APIs - [Enterprise Platform Development](/writings/enterprise-platform-development-telecommunications-advertising/) - Telecommunications and advertising technology at scale ## Conclusion Working at Tyroo to build analytical systems in Core Java was a transformative experience. It demonstrated the power of Java in handling big data and real-time analytics in the fast-paced world of digital advertising. The systems we built not only solved immediate challenges in the adtech industry but also laid the groundwork for future innovations in data-driven advertising. As the advertising technology landscape continues to evolve, the foundational work we did at Tyroo in building robust, scalable analytical systems remains more relevant than ever. This project not only revolutionized how advertising data was processed and analyzed in India but also set new benchmarks for what was achievable in adtech using Core Java. --- *About the author: [Dipankar Sarkar](/about/) is a technology leader with expertise in Java development and big data analytics. At Tyroo, India's largest AdTech company, he built high-performance analytical systems processing massive advertising datasets. [View all posts](/writings/) | [Get in touch](/contact/)* --- ## Enterprise Platform Development: Scaling Telecommunications and Advertising Solutions URL: https://www.dipankar.name/writings/enterprise-platform-development-telecommunications-advertising/ Date: 2020-11-12 Tags: platform-architecture, scalability, telecommunications, advertising-platform, enterprise Dipankar Sarkar built Kirusa Voice SMS (250M users) and enhanced Clickable PPC platform (TechCrunch Top 50) - enterprise-scale telecommunications and advertising. In 2009, I had the opportunity to work on two groundbreaking enterprise platforms that showcased the breadth of challenges in building scalable systems across different domains. At Kirusa in New Delhi, I developed a voice communication platform reaching over 250 million users, while at Clickable in Gurgaon (a TechCrunch Top 50 company), I enhanced the security and scalability of their flagship PPC advertising management solution. These experiences demonstrated how fundamental principles of enterprise software development apply across telecommunications and advertising technology. ## Kirusa Voice SMS: Revolutionizing Mobile Communication ### The Vision Kirusa aimed to bridge the gap between voice and text communication, creating a solution that would be accessible to a wide range of users, including those who found typing challenging or preferred voice communication. The goal was to develop a product that telecom providers could easily integrate into their existing systems. ### Technical Overview **Core Technologies:** - **J2EE (Java 2 Enterprise Edition)**: The backbone of our application, providing a robust and scalable server-side platform. - **MySQL**: Our choice for database management, offering reliability and performance for handling millions of user records and messages. **Key Features Developed:** 1. **Voice Recording and Compression**: Implemented efficient algorithms for capturing and compressing voice messages to minimize data usage. 2. **SMS Integration**: Developed a system to seamlessly convert voice messages into SMS notifications, ensuring compatibility with non-voice SMS users. 3. **Telecom Integration Layer**: Created a flexible integration layer allowing easy deployment across different telecom providers' systems. 4. **User Management System**: Built a scalable user management system capable of handling millions of users across multiple telecom networks. 5. **Message Queueing and Delivery**: Implemented a robust queueing system to manage message delivery, ensuring reliability even under high load. ### Technical Challenges and Solutions **Challenge: Scalability** With a potential user base of over 250 million, scalability was paramount. **Solution**: We leveraged J2EE's clustering capabilities, implementing a horizontally scalable architecture. We used JMS (Java Message Service) for asynchronous processing of voice messages, allowing the system to handle spikes in usage efficiently. **Challenge: Cross-Platform Compatibility** The application needed to work across various mobile devices and operating systems. **Solution**: We developed a thin client application using J2ME (Java 2 Micro Edition), ensuring compatibility with a wide range of mobile devices prevalent at the time. For newer smartphones, we created platform-specific apps that interfaced with our J2EE backend. **Challenge: Low-Bandwidth Optimization** Many users in emerging markets had limited bandwidth. **Solution**: We implemented advanced voice compression algorithms, reducing the size of voice messages without significantly compromising quality. We also optimized our protocols to minimize data transfer between the mobile client and the server. **Challenge: Integration with Telecom Systems** Each telecom provider had unique systems and protocols. **Solution**: We developed a modular integration layer using J2EE's EJB (Enterprise JavaBeans) technology. This allowed us to create custom connectors for each telecom provider while maintaining a consistent core application logic. ### Implementation and Deployment 1. **Agile Development**: We adopted an agile methodology, allowing us to rapidly iterate and adapt to changing requirements from telecom partners. 2. **Rigorous Testing**: Implemented comprehensive unit testing using JUnit and integration testing to ensure reliability across different network conditions. 3. **Phased Rollout**: We started with smaller telecom providers, gathering real-world performance data before scaling up to larger networks. 4. **24/7 Monitoring**: Developed a robust monitoring system using tools like Nagios, allowing us to proactively address any issues in real-time. ### Impact and Legacy The Kirusa Voice SMS project had a significant impact on mobile communication: - Successfully deployed across 20 wireless telecom providers, reaching over 250 million users. - Increased accessibility of messaging services, particularly benefiting users with literacy challenges. - Set new standards for voice-text integration in mobile communications. - Paved the way for future voice-based messaging applications and services. ## Clickable: Enhancing PPC Management at Scale ### The Vision Clickable, recognized as a TechCrunch Top 50 company, was at the forefront of simplifying pay-per-click (PPC) advertising management across major networks. The company aimed to streamline PPC management for advertisers and agencies, providing a unified platform to manage campaigns across multiple advertising networks. The goal was to make the complex world of PPC advertising more accessible and efficient for businesses of all sizes. ### Technical Overview **Core Technologies:** - **.NET Platform**: The foundation of our application development - **C#**: The primary programming language for backend logic - **ASP.NET**: Used for building dynamic web pages and web applications - **MS SQL**: Our database management system for storing and retrieving vast amounts of advertising data **Key Areas of Focus:** 1. **Security Enhancement**: Implementing robust security measures to protect sensitive advertising data and user information. 2. **Scalability Improvements**: Enhancing the system's ability to handle growing numbers of users and increasing data volumes. 3. **Performance Optimization**: Improving the speed and efficiency of data processing and reporting. 4. **Internal Product Engineering**: Developing and refining internal tools to support the core product. ### Technical Challenges and Solutions **Challenge: Data Security** Protecting sensitive advertising data and user information was paramount. **Solution**: We implemented a multi-layered security approach: - Utilized ASP.NET's built-in security features for authentication and authorization. - Implemented encryption for sensitive data both at rest and in transit using the .NET Framework's cryptography classes. - Developed a comprehensive audit logging system to track all data access and modifications. **Challenge: Scalability for Growing Data Volumes** As Clickable's user base grew, the system needed to handle increasingly large datasets efficiently. **Solution**: We focused on database and application scalability: - Implemented database partitioning in MS SQL to manage large tables more effectively. - Developed a caching layer using ASP.NET's caching capabilities to reduce database load. - Utilized asynchronous programming patterns in C# to improve application responsiveness under high load. **Challenge: Cross-Network Data Integration** Integrating data from multiple advertising networks, each with its own format and API, was complex. **Solution**: We created a flexible data integration framework: - Developed a modular architecture using C# interfaces and abstract classes to standardize data handling across different networks. - Implemented an ETL (Extract, Transform, Load) process using SQL Server Integration Services (SSIS) for efficient data processing. **Challenge: Real-time Reporting** Users needed up-to-date performance data to make informed decisions quickly. **Solution**: We enhanced our reporting capabilities: - Implemented a real-time data processing pipeline using .NET's Task Parallel Library for concurrent data processing. - Developed a custom reporting engine using ASP.NET and C# that could generate complex reports on-the-fly. ### Implementation Approach 1. **Agile Methodology**: Adopted Scrum for iterative development and quick response to changing requirements. 2. **Code Quality**: Implemented rigorous code review processes and utilized static code analysis tools to maintain high code quality. 3. **Automated Testing**: Developed comprehensive unit tests using NUnit and integration tests to ensure reliability and catch regressions early. 4. **Continuous Integration**: Set up a CI/CD pipeline using Team Foundation Server (TFS) for automated building, testing, and deployment. ### Impact and Achievements Our work at Clickable had significant impacts: - Enhanced platform security, building trust with enterprise clients and protecting sensitive advertising data. - Improved system scalability, allowing Clickable to handle a 200% increase in data volume without performance degradation. - Reduced report generation time by 60%, providing users with near real-time insights into their PPC campaigns. - Streamlined the onboarding process for new advertising networks, reducing integration time by 40%. ## Common Lessons: Building Enterprise-Scale Solutions Working on both Kirusa Voice SMS and Clickable's PPC platform in 2009 revealed fundamental principles that apply across enterprise software development: 1. **Scalability as a Core Principle**: Whether handling 250 million telecom users or managing growing advertising data volumes, designing for horizontal scalability from the start proved critical. 2. **Modular Architecture**: Both projects benefited from modular designs - Kirusa's telecom integration layer and Clickable's cross-network data framework demonstrated the importance of extensibility. 3. **Security and Reliability**: Enterprise systems must prioritize security and reliability, whether protecting voice communications or sensitive advertising data. 4. **Real-time Processing**: Both platforms required efficient handling of real-time data - voice message queuing at Kirusa and real-time reporting at Clickable. 5. **Agile Methodology**: Iterative development and rapid adaptation to changing requirements proved essential in both telecommunications and advertising technology domains. ## Related Reading More enterprise and platform development work: - [Building Analytical Systems at Tyroo](/writings/building-analytical-systems-core-java-tyroo-adtech/) - AdTech analytics and big data processing - [Scalable E-Commerce Infrastructure](/writings/building-scalable-ecommerce-infrastructure-platform-migration/) - Platform migration and high-performance services - [Oracle Reports Plugin Development](/writings/innovating-oracle-reports-web-service-pds-plugin-development/) - Enterprise software internship - [Integrated Ad Platform at Nykaa](/writings/revolutionizing-ecommerce-integrated-ad-platform-social-commerce/) - Modern advertising platforms ## Conclusion Working on Kirusa Voice SMS and Clickable's PPC management platform in 2009 was a transformative experience that showcased the universal challenges of enterprise software development. These projects demonstrated that fundamental principles of scalability, security, and performance optimization apply across diverse domains - from telecommunications serving hundreds of millions of users to advertising technology platforms managing complex multi-network campaigns. At Kirusa, we created a scalable, reliable system that revolutionized how millions of people communicated, making mobile messaging accessible to users with diverse needs and capabilities. At Clickable, we built robust, secure solutions that simplified PPC management for businesses worldwide, setting new standards for what advertisers could expect from their management tools. Both experiences highlighted the importance of building systems with security and scalability as core principles, not afterthoughts. By leveraging the right technology stacks - J2EE/MySQL for telecommunications and .NET/C#/MS SQL for advertising technology - we created solutions that not only solved immediate challenges but also laid the groundwork for future innovations in their respective fields. As enterprise software continues to evolve, the lessons learned from these projects remain relevant: the need for flexible, secure, and high-performance solutions; the value of modular architecture; and the transformative power of well-designed software in making complex business processes more accessible and efficient. --- *About the author: [Dipankar Sarkar](/about/) is a technology leader with expertise in enterprise platform development across telecommunications and advertising technology. He built voice communication platforms serving 250+ million users at Kirusa and enhanced PPC management solutions at Clickable, a TechCrunch Top 50 company. [View all posts](/writings/) | [Get in touch](/contact/)* --- ## Shaping the Future of Content Sharing: Early Days at SlideShare URL: https://www.dipankar.name/writings/shaping-future-content-sharing-early-days-slideshare/ Date: 2012-12-07 Tags: startup-scaling, ruby-on-rails, platform-architecture, team-building, startup-journey Dipankar Sarkar joined SlideShare as one of the first five engineers, building core features for the platform that became 'YouTube of presentations' (acquired by LinkedIn). In 2007-2008, fresh out of university, I had the extraordinary opportunity to join SlideShare as one of its first five software engineers. This experience thrust me into the heart of a startup that would go on to revolutionize how people share and access professional content online. ## The SlideShare Vision SlideShare aimed to become the YouTube of slide presentations, creating a platform where professionals could easily upload, share, and discover presentations on any topic. Our goal was to build a robust, user-friendly platform that could handle a wide variety of content types and scale to millions of users. ## Technical Overview As one of the early engineers, I was involved in various aspects of the platform's development: ### Core Technologies - **Ruby on Rails**: Our primary web framework, chosen for its rapid development capabilities - **MySQL**: For robust database management - **FreeBSD**: As our server operating system - **Nginx and Apache**: For web serving and proxying - **Various supporting technologies**: Including Python, PHP, and more ### Key Features Developed 1. **URL Upload System**: Implemented a feature allowing users to upload presentations directly from a URL. 2. **Server Management**: Involved in managing and scaling our server infrastructure to handle growing traffic. 3. **Conversion Engine**: Played a crucial role in developing the system that converted various file formats into web-friendly presentations. ## Technical Challenges and Solutions ### Challenge: Handling Diverse File Formats Users needed to upload presentations in various formats, which then had to be converted for web viewing. **Solution**: - Developed a robust conversion engine using open-source tools like OpenOffice. - Implemented a queuing system for efficient processing of uploads. - Created fallback mechanisms to handle conversion errors gracefully. ### Challenge: Scaling for Rapid Growth As SlideShare's popularity grew, we needed to ensure the platform could handle increasing loads. **Solution**: - Implemented caching strategies using Memcached to reduce database load. - Optimized database queries and implemented database sharding as data volumes grew. - Utilized content delivery networks (CDNs) to serve static content efficiently. ### Challenge: Ensuring High Availability With a growing user base, minimizing downtime became crucial. **Solution**: - Implemented load balancing using Nginx to distribute traffic across multiple application servers. - Developed a robust monitoring system to quickly identify and address issues. - Created automated deployment scripts to streamline updates and reduce human error. ## Key Contributions and Learnings 1. **Full-Stack Development**: Gained experience across the entire stack, from front-end design to back-end architecture and server management. 2. **Scalability Mindset**: Learned to design and implement features with scalability in mind from the outset. 3. **Agile Development**: Embraced agile methodologies, learning to iterate quickly and respond to user feedback. 4. **Open Source Collaboration**: Actively engaged with open-source communities, both using and contributing to various projects. 5. **Performance Optimization**: Developed skills in identifying and resolving performance bottlenecks in a high-traffic web application. ## Impact and Legacy Being part of SlideShare's early team had a lasting impact: - Helped build a platform that would eventually host millions of presentations and reach over 80 million users monthly. - Contributed to features that became central to SlideShare's identity and success. - Gained invaluable experience in scaling a startup from its early stages to significant growth. - Played a role in shaping a platform that would later be acquired by LinkedIn, validating its impact in the professional content sharing space. ## Conclusion My time at SlideShare was a formative experience that shaped my career as a software engineer. It provided a unique opportunity to be part of building a platform from its early stages, tackling challenges of scale, and contributing to a product that would significantly impact how professionals share knowledge online. The lessons learned at SlideShare – about rapid development, scalability, user-centric design, and the power of open-source technologies – have remained relevant throughout my career. As the landscape of content sharing and professional networking continues to evolve, the foundational work we did at SlideShare stands as a testament to the power of innovative thinking and solid engineering in creating platforms that connect and empower users worldwide. This experience underscored the excitement and challenges of startup engineering, where limited resources must be balanced with ambitious goals, and where each team member's contributions can have a significant and lasting impact on the product's success. ## Related Reading More early startup work: - [Mobile Banking at MPower Money](/writings/revolutionizing-mobile-banking-phire-mpower-money-python-metaprogramming/) - Innovative FinTech solutions - [E-commerce Marketplace at Tradus](/writings/pioneering-indias-first-ecommerce-marketplace-tradus/) - Building scalable platforms --- *About the author: [Dipankar Sarkar](/about/) is a technology leader with startup engineering expertise. As one of SlideShare's first five engineers, he built core features for the platform that became the 'YouTube of presentations.' [View all posts](/writings/) | [Get in touch](/contact/)* --- ## Revolutionizing Mobile Banking: Developing PHIRE at MPower Money with Python and Metaprogramming URL: https://www.dipankar.name/writings/revolutionizing-mobile-banking-phire-mpower-money-python-metaprogramming/ Date: 2012-09-25 Tags: fintech, python, mobile-banking, startup-scaling, financial-inclusion Dipankar Sarkar built PHIRE at MPower Money - the world's first mobile debit network enabling SMS-based banking for unbanked populations using Python. In 2008-2009, as mobile technology was beginning to transform various industries, I had the opportunity to be part of a groundbreaking project at MPower Money in New Delhi, India. As a Software Engineer, I was instrumental in developing PHIRE, the world's first mobile debit network that allowed users to perform banking transactions via SMS, leveraging the power of Python and advanced metaprogramming techniques. ## The MPower Vision MPower Money aimed to revolutionize financial access in India, where a significant portion of the population was unbanked or underbanked. The goal was to create a system that would allow anyone with a basic mobile phone to perform banking transactions, effectively turning SMS into a financial tool. ## PHIRE: A Breakthrough in Mobile Banking PHIRE (Phone Initiated Remittance Engine) was designed to be a game-changing platform in the world of mobile banking. It allowed users to: 1. Check account balances 2. Transfer money to other users 3. Pay bills 4. Recharge mobile credits 5. Receive notifications for transactions All of these functions could be performed using simple SMS commands, making banking accessible to anyone with a mobile phone, regardless of internet connectivity or smartphone ownership. ## Technical Overview ### Core Technologies - **Python**: The primary programming language for the entire platform development - **Metaprogramming**: Extensively used for creating flexible and dynamic code structures - **SQLite**: For lightweight, serverless database management - **SMS Gateway Integration**: To handle incoming and outgoing SMS messages ### Key Features Developed 1. **Dynamic SMS Command Parsing**: Utilized metaprogramming to create a flexible system for interpreting and executing SMS commands. 2. **Secure Transaction Processing**: Implemented a secure system for processing financial transactions initiated via SMS, using Python's cryptography libraries. 3. **Real-time Balance Updates**: Ensured that account balances were updated in real-time after each transaction, leveraging Python's asynchronous capabilities. 4. **Banking System Integration**: Developed Python interfaces to connect with existing banking infrastructure. 5. **Automated Response System**: Created a system to send automated SMS responses for transaction confirmations and account inquiries, using Python's string formatting capabilities. ## Technical Challenges and Solutions ### Challenge: Creating a Flexible Command System We needed a system that could easily adapt to new banking features and commands. **Solution**: - Implemented an extensive metaprogramming framework in Python, allowing for dynamic creation and modification of banking commands. - Used Python decorators and metaclasses to create a declarative syntax for defining new SMS commands, making it easy for developers to add new functionality. ### Challenge: Ensuring Security in SMS Banking Securing financial transactions conducted via SMS was paramount. **Solution**: - Developed a custom encryption system using Python's cryptography libraries. - Implemented a two-factor authentication system using one-time passwords (OTP) generated through Python's secure random number generation. - Created a fraud detection algorithm using machine learning libraries in Python to identify and flag suspicious transaction patterns. ### Challenge: Handling High Volume of SMS Transactions The system needed to process a large number of SMS messages quickly and accurately. **Solution**: - Leveraged Python's asyncio library to handle multiple SMS messages concurrently. - Implemented a custom message queuing system in Python to manage peak loads efficiently. - Used SQLite with optimized indexing for fast transaction processing and efficient storage. ### Challenge: Ensuring Transaction Reliability Given the sometimes unreliable nature of SMS delivery, ensuring transaction reliability was crucial. **Solution**: - Developed a robust transaction management system with rollback capabilities using Python's context managers. - Implemented a message acknowledgment system using coroutines to confirm the receipt and processing of each SMS command. - Created a transaction reconciliation process using Python's threading module to handle cases of network failures or delayed messages. ## Implementation and Impact 1. **Rapid Development**: Python's flexibility and the use of metaprogramming allowed for quick iterations and feature additions. 2. **Scalability**: The system was able to handle millions of transactions monthly, showcasing the scalability of our Python-based architecture. 3. **Financial Inclusion**: PHIRE played a significant role in bringing basic banking services to underserved populations, particularly in rural areas. 4. **Innovation in FinTech**: The use of Python and metaprogramming in a financial system was pioneering at the time, setting new standards in FinTech development. ## Conclusion Working on PHIRE at MPower Money was a transformative experience that showcased the potential of Python and metaprogramming in creating innovative financial solutions. By leveraging these technologies, we were able to create a highly flexible and scalable system that brought banking services to millions who previously had limited or no access to traditional banking. This project was at the forefront of the mobile banking revolution, demonstrating that with creative use of programming paradigms like metaprogramming, it's possible to create solutions that have a profound impact on financial inclusion and accessibility. The success of PHIRE highlighted the importance of choosing the right technology stack and programming paradigms when designing solutions for complex problems. As mobile technology and programming languages continue to evolve, the principles of flexibility, security, and simplicity that we championed in PHIRE using Python and metaprogramming remain crucial in developing inclusive financial technologies. ## Related Reading More innovative platform work: - [Enterprise Platform Development](/writings/enterprise-platform-development-telecommunications-advertising/) - Telecommunications and advertising platforms at scale - [SlideShare Early Days](/writings/shaping-future-content-sharing-early-days-slideshare/) - Building content platforms --- *About the author: [Dipankar Sarkar](/about/) is a technology leader specializing in Python development and FinTech. At MPower Money, he developed PHIRE, the world's first mobile debit network, pioneering financial inclusion through SMS technology. [View all posts](/writings/) | [Get in touch](/contact/)* --- ## Early Career Open Source Journey: Three Pioneering Projects (2005-2006) URL: https://www.dipankar.name/writings/early-career-open-source-journey-pioneering-projects/ Date: 2011-11-28 Tags: open-source, google-summer-of-code, mozilla, linux-kernel, startup-journey Dipankar Sarkar was one of the first five Indians in Google Summer of Code 2005, building WYSIWYG XUL editor for Mozilla and NFSv4 testing frameworks. In 2005-2006, as a budding software engineer, I had extraordinary opportunities to contribute to the open-source community through Google Summer of Code and internships. These three formative projects - developing a WYSIWYG XUL editor for Mozilla, building a Personal Video Recording prototype, and creating an NFSv4 testing framework - provided invaluable insights into collaborative development, embedded systems, and distributed systems testing. ## 1. WYSIWYG XUL Editor for Mozilla (Google Summer of Code 2005) ### Project Overview In 2005, I had the extraordinary opportunity to be one of the first five Indians selected to participate in the inaugural Google Summer of Code program. My project involved working with Mozilla, focusing on developing a WYSIWYG (What You See Is What You Get) editor for XUL (XML User Interface Language), a markup language used for creating Mozilla's user interfaces. The main objective was to create a user-friendly, visual editor for XUL. This tool would significantly simplify the process of creating user interfaces for Mozilla applications, making it more accessible to developers who were not deeply familiar with XUL syntax. ### Technical Approach **Technologies and Tools Used:** - **XUL**: The XML-based language for Mozilla's user interface - **JavaScript**: For implementing editor functionality - **DOM (Document Object Model)**: For manipulating XUL elements - **CSS**: For styling the editor interface and XUL output - **Chameleon**: An existing Mozilla project that served as the initial codebase **Key Components Developed:** 1. **Visual Editing Interface**: - Developed a drag-and-drop interface for XUL elements - Implemented real-time preview of XUL layouts 2. **XUL Element Library**: - Created a comprehensive library of XUL elements that users could easily insert into their designs 3. **Property Editor**: - Built a system for editing properties of XUL elements visually 4. **Code Generation**: - Implemented functionality to generate clean, well-formatted XUL code from the visual design 5. **Integration with Mozilla Framework**: - Ensured the editor worked seamlessly within the Mozilla development environment ### Challenges and Solutions **Challenge: Understanding XUL and Mozilla's Codebase** As a newcomer to Mozilla development, understanding XUL and navigating Mozilla's extensive codebase was initially daunting. **Solution**: Engaged deeply with Mozilla documentation, participated actively in developer forums, and sought guidance from Mozilla mentors. **Challenge: Balancing Visual Editing with Code Fidelity** Creating a WYSIWYG editor that produced clean, efficient XUL code was challenging. **Solution**: Implemented a robust code generation system that prioritized readability and efficiency. Included options for advanced users to fine-tune the generated code. **Challenge: Cross-Platform Compatibility** Ensuring the editor worked consistently across different operating systems was crucial. **Solution**: Leveraged Mozilla's cross-platform framework and conducted extensive testing on various operating systems to ensure compatibility. ### Impact and Contributions 1. **Simplified XUL Development**: The editor made XUL interface development more accessible to a broader range of developers. 2. **Community Engagement**: The project fostered increased interest and participation in Mozilla's open-source community. 3. **Innovation in Tool Development**: Set a precedent for creating visual development tools within the Mozilla ecosystem. 4. **Code Contribution**: The project's codebase was contributed back to the Mozilla community, serving as a foundation for future XUL development tools. ## 2. Personal Video Recording Prototype (Tekriti Software, Summer 2005) ### Project Overview In the summer of 2005, I had the opportunity to intern at Tekriti Software in Gurgaon, India. This internship provided me with a unique challenge: to develop a Personal Video Recording (PVR) prototype, similar to TiVo, using off-the-shelf hardware and open-source software. This project was at the forefront of the digital home entertainment revolution, offering valuable insights into embedded systems and open-source development. The main objective was to create a functional PVR system that could: 1. Record live TV 2. Provide an interactive program guide 3. Offer basic playback controls (pause, rewind, fast-forward live TV) 4. Operate on affordable, readily available hardware ### Technical Approach **Hardware and Software Stack:** - **Hardware**: VIA EPIA embedded board (a compact, low-power x86 platform) - **Operating System**: Linux (customized distribution) - **PVR Software**: MythTV (open-source PVR software suite) - **Programming Languages**: C++ (for MythTV customizations), Python (for web crawling and data processing) - **Database**: MySQL (for storing program information) **Key Components Developed:** 1. **Custom Linux Build**: - Created a streamlined Linux distribution optimized for the VIA EPIA board - Configured the system for diskless boot to minimize moving parts 2. **MythTV Integration**: - Compiled and optimized MythTV for the embedded platform - Customized the MythTV interface for better usability on a TV screen 3. **Electronic Program Guide (EPG) Generation**: - Developed a Python-based web crawler to extract TV listings from Indiatimes.com - Created a parser to convert the crawled data into a format compatible with MythTV's EPG system 4. **Hardware Integration**: - Configured TV tuner cards to work with the VIA EPIA board - Implemented drivers for remote control functionality 5. **Performance Optimization**: - Fine-tuned the system for optimal performance on limited hardware resources - Implemented efficient video encoding and storage mechanisms ### Challenges and Solutions **Challenge: Limited Hardware Resources** The VIA EPIA board had constrained processing power and memory compared to full-fledged PCs. **Solution**: Optimized the Linux build by removing unnecessary components and fine-tuning MythTV's configuration for low-resource environments. Implemented efficient buffering and caching mechanisms. **Challenge: Reliable EPG Data** Consistently obtaining accurate and up-to-date program information was crucial for the PVR's functionality. **Solution**: Developed a robust web crawling system with error handling and redundancy. Implemented a local caching system to ensure EPG availability even during internet outages. **Challenge: User Interface for TV** Designing a user interface that was easily navigable on a TV screen with a remote control posed unique challenges. **Solution**: Customized MythTV's interface, emphasizing large, clear fonts and simplified navigation suitable for remote control use. Conducted usability testing with potential users to refine the interface. ### Open Source Contributions A significant aspect of this project was its commitment to open source: 1. **Code Contributions**: Parts of the customized MythTV code and the EPG crawler were contributed back to the open-source community. 2. **Documentation**: Created detailed documentation of the build process and customizations, making it easier for others to replicate or build upon our work. 3. **Community Engagement**: Actively participated in MythTV and Linux embedded system forums, sharing insights and seeking community input. ## 3. NFSv4 Testing Framework (OSDL, Google Summer of Code 2006) ### Project Overview In the summer of 2006, I had the exciting opportunity to participate in the Google Summer of Code program again, this time working with the Open Source Development Labs (OSDL). My project focused on improving the testing infrastructure for NFSv4 (Network File System version 4), a crucial component in distributed file systems. This experience not only enhanced my technical skills but also introduced me to the world of distributed systems testing. The main objective was to develop a comprehensive testing framework for NFSv4, leveraging network emulation capabilities provided by the Linux kernel. This involved: 1. Creating a suite of testing scripts for NFSv4. 2. Integrating these scripts with NetEm, a network emulation tool in the Linux kernel. 3. Enhancing OSDL's ability to thoroughly test NFSv4 under various network conditions. ### Technical Approach **Tools and Technologies Used:** - **Bash Scripting**: Primary language for developing test scripts. - **Python**: Used for more complex test scenarios and data analysis. - **NetEm**: Linux kernel's network emulation tool for simulating various network conditions. - **NFSv4**: The target file system protocol being tested. - **Linux Kernel**: The environment for both NFSv4 and NetEm. **Key Components Developed:** 1. **Test Script Suite**: - Developed a comprehensive set of Bash and Python scripts to test various aspects of NFSv4. - Covered scenarios like file operations, locking mechanisms, and performance under different loads. 2. **NetEm Integration**: - Implemented scripts to configure NetEm for simulating diverse network conditions. - Simulated scenarios like high latency, packet loss, and bandwidth limitations. 3. **Automated Testing Framework**: - Created a framework to automate the execution of tests under different network conditions. - Implemented logging and result analysis features for easy interpretation of test outcomes. 4. **Documentation**: - Wrote detailed documentation for the testing framework and individual test cases. - Created user guides for OSDL team members to easily run and extend the tests. ### Challenges and Solutions **Challenge: Understanding NFSv4 Intricacies** NFSv4 is a complex protocol with many nuances. **Solution**: Engaged in extensive reading of NFSv4 specifications and discussions with OSDL mentors to gain a deep understanding of the protocol. **Challenge: Simulating Real-World Network Conditions** Creating realistic network scenarios for testing was crucial but challenging. **Solution**: Leveraged NetEm's capabilities extensively, researching and implementing configurations that closely mimicked real-world network behaviors. **Challenge: Ensuring Test Reliability** Ensuring that tests were reliable and reproducible across different environments was important. **Solution**: Implemented rigorous error checking and environment validation in the test scripts. Also, created a standardized testing environment specification. ### Impact and Contributions 1. **Improved Testing Efficiency**: The automated test suite significantly reduced the time and effort required for NFSv4 testing at OSDL. 2. **Enhanced Test Coverage**: The integration with NetEm allowed OSDL to test NFSv4 under a wide range of network conditions, improving overall reliability. 3. **Open Source Contribution**: The tools and scripts developed were contributed back to the open-source community, benefiting other developers and organizations working with NFSv4. 4. **Knowledge Sharing**: The documentation and guides created helped in knowledge transfer and made it easier for new contributors to understand and work on NFSv4 testing. ## Personal Growth and Learning These three projects provided invaluable learning across multiple dimensions: 1. **Open Source Collaboration**: Gained valuable experience in contributing to major open-source projects and collaborating with global communities of developers. 2. **Full-Stack Development**: Developed skills across the entire stack, from low-level system optimization to user interface design. 3. **Embedded Systems Expertise**: Gained hands-on experience in developing for resource-constrained embedded environments. 4. **Testing Methodologies**: Developed a strong understanding of software testing principles, particularly for distributed systems. 5. **Linux Kernel Familiarity**: Gained valuable exposure to Linux kernel internals, especially in networking and file systems. 6. **Community Interaction**: Learned the importance of community engagement and open communication in open-source development. ## Historical Context and Legacy It's important to note that in 2005-2006, these technologies were at the forefront of their respective domains. While XUL has since been phased out in favor of more modern web technologies, and dedicated PVR devices have been largely superseded by integrated smart TV functions and streaming services, the skills and concepts learned during these projects laid a strong foundation for modern development practices. The principles of: - Creating intuitive, visual tools for developers - System integration and user experience design - The potential of open-source software in consumer electronics - Robust testing in developing reliable software systems - The growing convergence of traditional technologies and internet-based solutions All remain relevant and continue to inform modern software development. ## Conclusion My participation in the Google Summer of Code program and internship at Tekriti Software were landmark experiences in my early career. Being one of the first five Indians selected for Google Summer of Code 2005 was not only an honor but also a significant responsibility. These projects allowed me to contribute meaningfully to the open-source community and helped shape my understanding of collaborative software development. The experiences of working on a WYSIWYG XUL editor, building a PVR prototype, and developing an NFSv4 testing framework were invaluable, providing insights into user interface design, embedded systems, distributed systems testing, and the challenges of creating developer tools and consumer electronics. Although some of these technologies have evolved or been superseded, the foundational skills and open-source contribution mindset developed during this period continued to influence my career path, reinforcing the importance of innovation, collaboration, and user-centric design in software development. As the landscape of software development continues to evolve, the lessons learned from these early career projects - about system integration, community collaboration, testing rigor, and the power of open-source development - remain relevant, continually informing my approach to technology development and innovation. ## Related Reading More early career and enterprise work: - [Oracle Reports Plugin Development](/writings/innovating-oracle-reports-web-service-pds-plugin-development/) - Enterprise software internship - [Computer Vision at B-Core Tokyo](/writings/optimizing-vision-algorithms-research-bcore-software-tokyo/) - International research experience - [Machine Learning at Hike](/writings/machine-learning-hike-ai-systems/) - Leading ML team serving millions --- *About the author: [Dipankar Sarkar](/about/) is a technology leader passionate about open-source development. As one of the first five Indians in Google Summer of Code 2005, he developed a WYSIWYG XUL editor for Mozilla and contributed to NFSv4 testing frameworks, shaping his approach to collaborative software development. [View all posts](/writings/) | [Get in touch](/contact/)* --- ## Optimizing Vision Algorithms: My Research Experience at B-Core Software in Tokyo URL: https://www.dipankar.name/writings/optimizing-vision-algorithms-research-bcore-software-tokyo/ Date: 2010-11-15 Tags: computer-vision, algorithm-optimization, image-processing, research Dipankar Sarkar optimized computer vision algorithms at B-Core Software Tokyo, gaining international experience in image processing and algorithm optimization. In 2007, fresh out of university, I had the unique opportunity to work as a Researcher and Software Developer at B-Core Software Private Limited in Tokyo, Japan. This experience not only broadened my technical skills but also provided valuable insights into the Japanese approach to software development and research. ## The B-Core Vision B-Core Software specializes in developing cutting-edge computer vision solutions. My role involved delving deep into vision algorithms and software optimization techniques, with a focus on understanding and improving complex software specifications. ## Research Focus and Challenges ### Understanding Vision Algorithms My primary task was to analyze and optimize various computer vision algorithms. This involved: 1. Studying state-of-the-art vision algorithms, including edge detection, feature extraction, and image segmentation. 2. Analyzing the performance bottlenecks in existing implementations. 3. Proposing and implementing optimizations to improve algorithm efficiency. ### Software Specification Analysis A significant part of my work involved understanding and improving software specifications. This included: 1. Analyzing detailed software requirements and architecture documents. 2. Identifying areas where specifications could be improved for better implementation and performance. 3. Collaborating with Japanese colleagues to bridge the gap between specification and implementation. ## Technical Approach ### Tools and Technologies - **C++**: The primary language for implementing and optimizing vision algorithms. - **OpenCV**: Leveraged for its comprehensive computer vision library. - **MATLAB**: Used for rapid prototyping and algorithm visualization. - **Linux**: The primary development environment. ### Optimization Techniques 1. **Algorithm Refinement**: Improved existing algorithms by reducing computational complexity. 2. **Memory Optimization**: Implemented techniques to reduce memory usage in vision processing pipelines. 3. **Parallelization**: Explored ways to parallelize algorithms for multi-core processors. 4. **SIMD Instructions**: Utilized Single Instruction Multiple Data (SIMD) instructions for performance boost. ## Cultural and Professional Insights Working in Tokyo provided unique insights into Japanese work culture and software development practices: 1. **Attention to Detail**: Learned the importance of meticulous documentation and specification. 2. **Collaborative Problem-Solving**: Experienced the Japanese approach to group problem-solving and consensus-building. 3. **Long-Term Thinking**: Observed how Japanese companies invest in research with a long-term perspective. ## Challenges and Learning ### Language Barrier While technical documents were in English, day-to-day communication was challenging. **Solution**: Took basic Japanese language classes and relied on visual communication tools for complex ideas. ### Different Approach to Specifications Japanese software specifications were more detailed and rigid compared to what I was used to. **Solution**: Adapted to the Japanese style of comprehensive documentation while suggesting areas where flexibility could improve efficiency. ## Impact and Takeaways 1. **Technical Growth**: Gained deep insights into computer vision algorithms and optimization techniques. 2. **Cross-Cultural Experience**: Developed an appreciation for different approaches to software development. 3. **Research Skills**: Enhanced my ability to read and understand complex technical specifications. 4. **Global Perspective**: Gained a broader perspective on the global tech industry. ## Conclusion My experience at B-Core Software in Tokyo was transformative, both professionally and personally. It laid a strong foundation for my career in software development and research, especially in the field of computer vision. The skills I acquired in algorithm optimization and the insights into meticulous software specification practices have been invaluable throughout my career. This experience underscored the importance of global exposure in the tech industry. It taught me that diversity in approach and thinking is crucial for innovation in software development. As the field of computer vision continues to evolve, the lessons learned from this early career experience in Japan continue to influence my approach to problem-solving and innovation in technology. ## Related Reading More computer vision work: - [Machine Learning at Hike](/writings/machine-learning-hike-ai-systems/) - Advanced computer vision for personalized avatars and AI systems - [AutoInspect and AutoSpray](/writings/autoinspect-autospray-ml-driven-precision-industrial-robotics/) - ML-driven industrial vision systems --- *About the author: [Dipankar Sarkar](/about/) is a technology leader specializing in computer vision and algorithm optimization. His international research experience at B-Core Software in Tokyo shaped his approach to image processing and software specification. [View all posts](/writings/) | [Get in touch](/contact/)* --- ## Innovating Oracle Reports: Developing a Web Service PDS Plugin at Oracle Corporation URL: https://www.dipankar.name/writings/innovating-oracle-reports-web-service-pds-plugin-development/ Date: 2010-03-10 Tags: java, enterprise-software, web-services, plugin-architecture Dipankar Sarkar developed a Web Service PDS plugin for Oracle Reports during internship at Oracle Corporation, receiving recognition from US headquarters. In 2006, during my undergraduate studies, I had the invaluable opportunity to intern at Oracle Corporation in Bengaluru, India. This experience not only exposed me to enterprise-level software development but also allowed me to contribute significantly to Oracle's reporting solutions. My primary project involved developing a Web Service PDS (Pluggable Data Source) plugin for the Oracle Reports server, a task that would enhance the capabilities of this widely-used enterprise reporting tool. ## Project Overview The main objective of my internship project was to create a plugin that would allow Oracle Reports to consume data from web services, expanding its data sourcing capabilities. This plugin would enable Oracle Reports to integrate seamlessly with modern web-based data sources, enhancing its relevance in an increasingly service-oriented architectural landscape. ## Technical Approach ### Technologies and Tools Used - **Java**: The primary programming language for plugin development - **Oracle Reports**: The target platform for the plugin - **Web Services**: SOAP and early RESTful services - **XML**: For data representation and configuration - **JDBC**: For database interactions within Oracle's ecosystem - **Eclipse IDE**: The development environment ### Key Components Developed 1. **Web Service Connector**: - Developed a robust connector to interface with various web services - Implemented support for both SOAP and REST protocols - Created a flexible configuration system for easy setup of web service endpoints 2. **Data Transformation Layer**: - Built a system to transform web service responses into a format compatible with Oracle Reports - Implemented XML parsing and data mapping functionalities 3. **Pluggable Data Source Architecture**: - Designed the plugin to adhere to Oracle's PDS architecture - Ensured seamless integration with existing Oracle Reports workflows 4. **Caching Mechanism**: - Implemented an intelligent caching system to optimize performance for frequently accessed data 5. **Error Handling and Logging**: - Developed comprehensive error handling to manage web service failures gracefully - Created detailed logging for troubleshooting and performance monitoring ## Challenges and Solutions ### Challenge: Understanding Oracle's Complex Ecosystem As an intern, grasping Oracle's extensive and complex software ecosystem was initially overwhelming. **Solution**: Engaged in intensive study of Oracle documentation, participated in internal training sessions, and sought guidance from experienced mentors within the team. ### Challenge: Ensuring Cross-Version Compatibility The plugin needed to work across different versions of Oracle Reports. **Solution**: Implemented version checking and adaptive coding practices to ensure compatibility. Extensively tested the plugin on multiple versions of Oracle Reports. ### Challenge: Performance Optimization Integrating web services had the potential to slow down report generation. **Solution**: Developed an efficient caching mechanism and implemented asynchronous data fetching where possible to minimize impact on report generation time. ## Impact and Recognition 1. **Enhanced Functionality**: The plugin significantly expanded Oracle Reports' data sourcing capabilities, allowing it to integrate with modern web-based systems. 2. **Positive Feedback**: The plugin received appreciation and positive reviews from Oracle's US headquarters, validating its utility and quality. 3. **Potential for Product Integration**: There were discussions about incorporating the plugin into future Oracle Reports releases, highlighting its value to the product. 4. **Knowledge Transfer**: Created comprehensive documentation and conducted a knowledge transfer session, ensuring the team could maintain and extend the plugin after my internship. ## Personal Growth and Learning 1. **Enterprise Software Development**: Gained invaluable exposure to enterprise-level software development practices and standards. 2. **Java Proficiency**: Significantly enhanced my Java programming skills, particularly in areas of enterprise application development. 3. **Web Services Understanding**: Developed a deep understanding of web services and their integration with enterprise systems. 4. **Professional Work Environment**: Experienced working in a professional, multinational corporate environment for the first time. ## Conclusion My internship at Oracle Corporation was a defining experience in my early career. Developing the Web Service PDS plugin for Oracle Reports not only allowed me to contribute to a major enterprise software product but also provided me with insights into the complexities and challenges of enterprise software development. The success of this project, evidenced by the positive reception from Oracle's US headquarters, was a significant boost to my confidence as a budding software developer. It demonstrated the impact that innovative thinking and solid development practices could have, even in a large, established product like Oracle Reports. This experience laid a strong foundation for my future work in software development, particularly in areas of enterprise solutions and system integration. The skills and knowledge gained during this internship at Oracle have continued to influence my approach to software development throughout my career, emphasizing the importance of creating flexible, efficient, and user-focused solutions in enterprise environments. ## Related Reading More enterprise development work: - [Enterprise Platform Development](/writings/enterprise-platform-development-telecommunications-advertising/) - Telecommunications and advertising platforms - [Analytical Systems at Tyroo](/writings/building-analytical-systems-core-java-tyroo-adtech/) - Enterprise-scale Java development --- *About the author: [Dipankar Sarkar](/about/) is a technology leader with enterprise software expertise. During his internship at Oracle Corporation, he developed innovative plugins for Oracle Reports, earning recognition from Oracle's US headquarters. [View all posts](/writings/) | [Get in touch](/contact/)* --- ## Contact - Website: https://www.dipankar.name - Email: contact@dipankar.name - LinkedIn: https://www.linkedin.com/in/dipankarsarkar - ORCID: https://orcid.org/0000-0001-5431-6367