Drivers of Inevitable Evolution
When Tactical Scripts Become Production Infrastructure
A Python script scheduled on cron, a hastily assembled Airflow DAG, or a notebook running overnight. These all begin their lives as expedient solutions to immediate problems. Within months, dozens of downstream systems depend on their output. Within a year, they will underpin executive dashboards and regulatory reporting. The pipeline that nobody planned to maintain has become infrastructure that nobody can afford to lose.
This reflection was inspired by a Reddit thread I saw a few weeks ago. The pattern it surfaced is neither novel nor surprising to practitioners, yet it remains persistently underestimated in its inevitability and impact. Organizations repeatedly discover that their most essential data infrastructure emerged through pragmatic necessity rather than deliberate design, evolving from a temporary workaround to a foundational dependency through dynamics that appear organic yet follow recognizable patterns.
Dependency accumulation occurs through diffusion rather than through decision making. Each incremental adoption feels reasonable in isolation: a dashboard here, an automation there, a downstream system needing yesterday’s output. The architecture diagram, if one exists, fails to capture the web of implicit contracts the pipeline now fulfills. Refactoring requires navigating not just technical complexity but organizational politics, as any proposed change must now satisfy stakeholders who never formally approved the original implementation.
What you may have missed:
👋🏿 Hi, I’m Hodman, and I help companies build reliable data infrastructure.
Earlier this week, I published a testing framework for data pipelines that covers business logic tests, data quality checks, statistical drift detection, and golden dataset validation. Most teams test their code thoroughly but miss the environmental failures that cause production incidents: schema changes, late data, and volume anomalies. You can read about it here.
Understanding why tactical pipelines become permanent fixtures requires examining the organizational and technical forces that enable their evolution.
Conway’s Law states that organizations design systems reflecting their communication structures. The corollary relevant here is that systems evolve to fill organizational voids. When formal data infrastructure teams lack the capacity or mandate to address emerging analytical needs, individuals solve problems locally using available tools and permissions. These solutions succeed precisely because they bypass governance overhead and lengthy provisioning cycles.
Business velocity amplifies this pattern. Organizations prioritizing speed-to-insight reward engineers who deliver working solutions over those who architect sustainable ones. The incentive structure favors demonstrating value over investing in operationalization. A pipeline demonstrating immediate ROI receives implicit authorization to persist, while requests for engineering time to rebuild it properly compete against new feature development.
Ward Cunningham coined the technical debt metaphor in 1992 to describe how shortcuts create future obligations. The problem compounds when organizations neither track the debt they’ve incurred nor budget time to repay it, even as expanding dependencies raise interest rates.
Institutional knowledge atrophy accelerates the risk. Documentation typically remains sparse or absent, reflecting the pipeline’s presumed temporary nature. When the original engineer moves on, the system becomes opaque to newcomers who must maintain it without understanding its full behavior or downstream contracts. Modifications carry unknown risks, creating a paradox: the pipeline demands continuous operation yet resists change because no one possesses complete knowledge of its function or dependencies.
Consequences of Unmanaged Evolution
Systems designed for occasional batch jobs begin running under production SLAs they were never architected to meet. Error handling extends only to retry logic and manual intervention. Monitoring tracks job completion but not data quality or business logic correctness. Without runbooks, redundancy, or rollback procedures, incident response becomes improvisational.
The same agility that initially made the pipeline attractive becomes a constraint. Schema changes that should take hours require cross-functional negotiation. Technology upgrades demand regression testing across unknown dependencies. New capabilities must work around legacy assumptions. Engineers proposing improvements discover that tactical infrastructure now dictates strategic boundaries.
Senior engineers gravitate toward building new systems rather than maintaining fragile legacy ones. Organizations often staff these pipelines with junior engineers, treating them as simple maintenance work. This understaffing creates a knowledge vacuum that cannot be rapidly filled when problems emerge, particularly after experienced personnel have moved to other roles.
The platform fragments as new tactical solutions proliferate. Building something fresh often appears more tractable than understanding and extending existing systems. Each solution introduces distinct technologies, failure modes, and operational requirements. Infrastructure costs scale with pipeline count, while engineering efficiency decreases as context switching overhead increases.
Governing Foundational Asset
Organizations need explicit criteria for when pipelines transition from experimental to production status. These thresholds should trigger mandatory investment in observability, documentation, and architectural review through policy enforcement integrated into development workflows and resource allocation, not through voluntary compliance.
Platform engineering teams can maintain registries of tactical solutions, assessing dependency footprint and architectural risk quarterly. Engineering leadership should allocate capacity for debt repayment the same way they budget for security patching: as mandatory maintenance rather than discretionary work competing with feature development.
Sustainability becomes easier when well-patterned templates, shared libraries, and platform capabilities make building maintainable solutions as expedient as building fragile ones. When the path of least resistance produces observable, documented, and testable pipelines, engineers under time pressure create more sustainable artifacts by default.
Every pipeline should include decommissioning criteria and retirement timelines from inception. Regular reviews can identify systems whose strategic value no longer justifies operational cost, creating permission to deprecate rather than accumulate indefinitely. Without sunset protocols, platforms become archaeological sites preserving every historical solution in production.
The phenomenon of tactical infrastructure becoming a strategic constraint reflects not individual engineering failure but systemic organizational dynamics. Recognition of these dynamics enables proactive governance rather than reactive crisis management. Data platforms that successfully scale do so not by preventing expedient solutions but by establishing clear pathways for tactical work to mature into sustainable infrastructure when business value justifies the investment.
Worth Reading
Several colleagues in the Substack data community published some great resources recently:
Charlotte Ledoux created a Data Advent Calendar for the holiday season at The Data Governance Playbook. It has attracted over 400 users and provides daily data governance insights.
Maggie @DataStoryteller wrote a comprehensive guide to data career specializations for anyone exploring the profession or considering a pivot within it.
Erfan Hesami published Data Quality Design Patterns at Pipeline to Insights, covering WAP, AWAP, TAP, and signal table patterns with implementation examples. The post offers practical methods for safeguarding production data, directly related to managing the tactical-to-foundational pipelines discussed here.
Alejandro Aboy built Substack Wrapped for authors to visualize their year of publishing. It has already processed over 900 publications in its first three days.
The "tactical to infrastructure" evolution nails a common data engineering trap. Making decommissioning criteria explicit from day one could prevent so many platforms becoming archaeological sites
Brilliant framing of how dependency accumlation happens through diffusion rather than deliberate design. The insight about staffing these "simple maintenance" pipelines with junior engineers while seniors move on is especially sharp becasue it creates exactly the knowledge vacuum that makes refactoring risky. The sunset protocol idea is practical, without retirement criteria from inception you really are just building an archaeological site one cron job at a time.