DATAAD EDITORIAL INTERVIEW | INDUSTRY VOICES
Interviewed and edited by JB Kim | DATAAD.COM
Editorial Introduction
For decades, the internet and the digital economy have been built upon an extraordinary ecosystem of shared software, open standards, independent developers, nonprofit foundations, and commercial technology companies.
Many of the tools we use every day — PHP, MySQL, GitHub, Apache software, and countless others — come from different organizations, yet work together to support an increasingly interconnected digital world.
Artificial intelligence is now introducing another fundamental change. Software development is no longer limited to people who can write every line of code themselves. AI assistants can help individuals develop websites, databases, business applications, and automated workflows.
However, the ability to create software raises a larger question: who is responsible for understanding, verifying, securing, and maintaining what AI helps us build?
To explore these issues, DATAAD invited Aditya Ranjan, a Senior Data Engineer with more than 16 years of experience in enterprise data engineering, distributed systems, and cloud data platforms, to participate in an editorial interview.
Beyond his professional experience, Ranjan has undertaken independent engineering investigations involving Apache Iceberg, Apache Spark, and other open-source data infrastructure. His public technical records address practical reliability questions such as signed-zero handling in query predicates, concurrency during lock-factory initialization, and floating-point row ordering. Through documented issues, proposed patches, regression tests, and public review discussions, these investigations illustrate the careful verification work behind dependable distributed systems.
Drawing on this experience, Ranjan discusses the evolution of open-source collaboration, the invisible work of software maintainers, the practical limits of AI-generated code, and the continuing importance of human accountability.
The following written interview presents eight questions from DATAAD and Ranjan's responses.
Q1. A Connected Ecosystem Without a Single Owner
DATAAD: To a reader familiar with PHP, MySQL, phpMyAdmin, VS Code, FileZilla, and GitHub, how would you explain the difference between open-source software, open standards, nonprofit foundations, and commercial platforms? How do these tools work together without belonging to one organization?
ADITYA RANJAN:
I find it easiest to explain by asking who controls what.
Open source is about the code itself: you can read it, run it and change it under a license. An open standard is a written specification that several independent products can implement, such as HTTP, SQL or the Parquet file format. A foundation like the Apache Software Foundation does not usually write the code. It gives projects a legal home, governance rules and continuity, so a project can outlive any one employer or maintainer. A commercial platform such as GitHub is a service run by a company. It hosts an enormous amount of open-source code, but hosting open source does not make the platform open source.
Take the small web stack in your question. PHP runs the application logic, MySQL holds the records, phpMyAdmin helps administer the database, VS Code is where the code gets edited, FileZilla moves files to the server and GitHub keeps the history. Nobody assembled these into one product. They fit together because they agree on protocols and formats: SQL, HTTP, SFTP, Git. I see the same thing in data platforms, only larger. Airflow schedules the work, Spark does the computation and Iceberg defines how tables are laid out on object storage. The pieces connect through documented interfaces, not through common ownership.
What I have learned over sixteen-plus years is that this openness has a cost, and the cost shows up when something breaks. You have to work out whether the bug belongs to the library, to a managed service, or to your own configuration. An open ecosystem lets you combine parts freely, but somebody on your team still has to own the combination.
Q2. From Traditional Databases to Distributed Data Infrastructure
DATAAD: What changed between the early web-server-and-database era and today's large-scale data infrastructure? Where do Apache Spark and Apache Iceberg fit, and what problems do they solve that traditional application databases such as MySQL were not designed to solve?
ADITYA RANJAN:
Early in my career, a big share of the work was moving data between operational databases, scheduled integration jobs and a reporting warehouse. Those systems were not primitive. They were designed for a different workload, and MySQL is still an excellent choice for many transactional applications. What changed is the volume and the variety. Organizations now keep years of history, event streams and mixed file formats, and they want to analyze all of it without slowing down the systems that serve customers.
Spark and Iceberg solve two different parts of that problem. Spark is the computation layer: it distributes transformations and analytical work across many machines. Iceberg is the table layer: it tracks snapshots, schema changes and metadata so that a very large set of files on object storage behaves like a table you can trust, including when several engines read and write it. Neither one replaces a transactional database, and neither will rescue a poorly designed data model.
A lesson I keep relearning is that adding compute is not the same as improving performance. Partitioning choices, shuffles, file sizes and data layout usually matter more than cluster size. The same instinct drives my independent work on storage-waste diagnostics for S3 and Delta Lake, which starts from a plain question: what is actually consuming the space, and can we account for it at the object and table level?
This matters more now because of AI. A model can produce a fluent answer from stale or inconsistent data, and the answer will look just as confident as a correct one. If we want AI applications that people can rely on, lineage, correctness and reproducibility in the data platform have to be treated as engineering requirements from the start.
Q3. The Invisible People Behind Open-Source Software
DATAAD: Who actually maintains and improves these widely used technologies? What motivates individual contributors and companies to invest in work that others can use freely, and what are the limits of that model?
ADITYA RANJAN:
Most people see the release announcement. Maintainers see the work between releases: reproducing a failure from a vague report, reading test logs, reviewing patches from strangers, deciding whether a change keeps compatibility, and sometimes saying no to a fix that looks reasonable. That work is mostly unglamorous and mostly unpaid or squeezed in around other duties.
From my own contributions to Apache Iceberg, Spark, Airflow and the dbt ecosystem, the most useful skill I have developed is making a problem reviewable by someone who never saw it happen. A small patch often needs a carefully chosen regression test and a clear explanation of why the old behavior was wrong. If the reviewer has to guess, the change usually stalls, and that is a fair outcome.
People contribute for mixed reasons. They need a bug fixed, they care about quality, they want to learn, or their company depends on the software. Companies gain when shared infrastructure gets more reliable. The limit of the model is that maintainer attention does not scale the way usage does. Security triage, compatibility testing and release management all take time, and a project can be used by thousands of organizations while being looked after by a handful of people.
AI-assisted coding sharpens this imbalance. Producing a patch is getting cheaper, but validating one is not. The best way I know to help a maintainer is to send fewer speculative changes and more reproducible evidence.
Q4. Engineering Reliability: Lessons from Apache Iceberg and Spark
DATAAD: Could you describe one or two specific contributions or engineering experiences involving Apache Spark or Apache Iceberg? What problem did you address, how was the work reviewed, and what did the process teach you? Public pull requests, issue links, or technical references would be helpful.
ADITYA RANJAN:
I will describe two Apache Iceberg cases. Both are behaviors that are easy to miss in routine testing but matter once a platform has more than one execution path.
Signed zero in predicates. IEEE 754 floating point has both +0.0 and -0.0. They compare as equal in ordinary arithmetic, but they are different at the representation level. When a query engine evaluates a filter through several paths, including Parquet dictionary filtering, inconsistent handling of the two values can change which rows come back. I worked on the predicate-evaluation issue (issue #18723, PR #18813) and on the related Parquet dictionary-filtering issue (issue #18886), including a focused regression test and a clean build of the affected module. The goal was not a clever change. It was to make the comparison semantics consistent across paths and to have a test that would catch a regression.
A possible initialization race in JdbcLockFactory. In issue #18797 and PR #18823, I looked at a possible race during initialization of Iceberg's JDBC lock factory. Concurrency defects are awkward because a normal sequential test can pass while the bug is still there. I proposed a fix together with a test written to exercise the failing interleaving. It reinforced a rule I follow: if I cannot explain how to reproduce a defect, I am not entitled to say I have fixed it.
Beyond Iceberg, I investigated floating-point row-ordering behavior in Spark (SPARK-60027) and worked on development-tooling changes connected to SPARK-60865. In Airflow, issue #76852 concerned unwanted debug output in an asynchronous S3Hook path, and I prepared a targeted change with a test. I have also contributed to the dbt ecosystem through dbt Hub PR #4841 (GitHub Actions checkout compatibility) and dbt-jsonschema PR #267 (test coverage for Jinja Boolean configuration).
On review: every one of these is public, with the discussion, the code and the maintainers' comments visible on the threads. I would rather readers check each record for its current status than rely on my summary, and I try to be careful to separate what I proposed, what I tested locally and what maintainers accepted. A submitted pull request is not proof of adoption.
What the cases share is attention to the edges: numeric representation, concurrent initialization, asynchronous execution and configuration parsing. Infrastructure assumptions tend to fail at those edges first, which is why I enjoy working there.
Q5. When AI Builds Software, Who Takes Responsibility?
DATAAD: A nonprogrammer can now ask AI to build a working CRM, website, or database application. What should such users understand about data models, dependencies, licensing, backups, updates, and access controls before deploying it for real users?
ADITYA RANJAN:
I am glad that people can now prototype software without spending years learning a language. I would still draw a firm line between "it runs on my laptop" and "I am ready to operate this for real users."
Start with the data model. The owner should be able to say where each important record lives, what makes it unique, how relationships are enforced and what happens when an update fails halfway through. If the answer is "the AI handled it," that is a warning sign. Next, list the packages and services the application depends on, check their licenses, and decide who will apply security updates and how often.
Backups are where demonstrations mislead most. Having a backup file is not the same as having proven you can restore it, so restore one into a clean environment before launch. Then treat authentication and authorization as separate questions. A user being able to log in does not mean that user should be able to read or edit every customer record.
Finally, make sure the operator can answer four plain questions after an incident: what changed, who accessed the data, what failed, and how do we roll back? If an AI agent can call tools or write to a database, give it the narrowest permissions that work and check what it actually did. AI can help build the application. It cannot take responsibility for the people whose information the application holds.
Q6. Security and Shared Responsibility
DATAAD: Recent security concerns involving widely used software such as PHP have renewed attention on maintenance and vulnerabilities. Without assuming any single project is representative, where does responsibility lie among maintainers, service providers, AI-tool makers, and application operators when a weakness is discovered?
ADITYA RANJAN:
I would not put the whole burden on an open-source maintainer, especially one working on personal time. In my view responsibility follows control. Maintainers can investigate reports, coordinate a fix and say clearly which versions are affected. Service providers are responsible for securing the environments they run. AI-tool vendors should reduce unsafe defaults and be honest about the limits of generated code. Application operators have to patch, set permissions, protect secrets and monitor what they deploy.
In data engineering I rarely see a vulnerability that belongs to a single component. More often it is an integration problem. A service can be implemented correctly and still be deployed with a role that has far more access than it needs. The exposure grows when AI agents can retrieve data and trigger actions across several systems, because each connection is another place where permissions can be too broad.
What I would like to see is a simple chain: a reproducible report, coordinated disclosure, a documented fix, clear upgrade guidance, and an operator who tests that the remediation actually worked. Security improves when each party can show what it did, instead of assuming another party took care of it.
Q7. Is AI Improving Open-Source Collaboration?
DATAAD: How is AI changing open-source contribution — in code generation, code review, vulnerability reporting, testing, and documentation? What genuinely improves collaboration, and what new problems might it create?
ADITYA RANJAN:
AI helps most when it shortens the time spent finding my way around unfamiliar code, setting up test scaffolding or summarizing a complicated change. I treat it as an aid. It does not replace understanding the behavior.
The signed-zero issue is a good example. A tool can suggest changing a comparison in seconds. The real question is whether that change preserves semantics across predicate evaluation and Parquet filtering, and answering it takes knowledge of the file format, a reproducible case, and tests that fail before the fix and pass after it.
The risk for open-source communities is a flood of plausible-looking contributions that move the hard work onto maintainers. A pull request with no clear failure case, no test and no explanation is not free, because someone else has to review it. Vulnerability reports have the same problem when nobody has validated them.
I would judge AI-assisted collaboration by the evidence it produces: smaller, focused changes, better tests, clearer documentation and more honest statements about what is still uncertain. Human accountability stays essential, particularly in infrastructure that other developers build on.
Q8. Is the Open-Source Model Ready for the AI Age?
DATAAD: For decades, independently organized open-source communities have contributed enormously to the digital world. Is their established model of collaboration sufficient for the AI era? If yes, what makes it resilient? If not, what concrete forms of cooperation should developers, foundations, companies, and AI providers consider — and who should take the lead?
ADITYA RANJAN:
The core model is sound. People who work for different organizations can agree on a technical problem, look at the evidence and improve shared software. I would protect that. What needs attention is the review and maintenance work around increasingly automated development.
I would prioritize three things. First, make reproducible tests and traceable changes a normal expectation, especially for correctness and security fixes. Second, treat maintainer time as essential infrastructure and fund it that way. Third, write down clear policies for AI-assisted contributions: provenance, who is responsible for review, and how unverified vulnerability reports are handled.
There is a wider AI-infrastructure question as well. In my article "The Hidden Infrastructure Costs of Autonomous AI: Why Cost per Verified Outcome Matters More Than Cost per Token," published on DATAAD on November 10, 2026, I argued that token counts alone are a poor measure of efficiency. Retrieval, tool calls, retries, orchestration and verification all consume resources, so the figure that matters is the cost of an outcome that has been verified. In my independent research on governance-as-code I have looked at the same problem from the control side: an AI system should leave evidence of what it was allowed to do and what it actually did.
On who should lead, I would not name one group. Foundations can offer neutral rules. Maintainers can define what counts as acceptable. Companies can contribute people and funding. AI providers can make their tools more transparent. Independent engineers can keep contributing focused fixes and published research.
After sixteen-plus years in data engineering, my view is simple. Systems become dependable when people are willing to investigate the unglamorous edge cases, and the AI era makes that habit more valuable, not less.
Public Technical References
The following public technical records were provided by Aditya Ranjan in connection with his independent engineering work. These references document issue investigations, proposed changes, tests, and public review discussions. Readers are encouraged to consult the original project records for their current status. A submitted issue or pull request does not necessarily indicate that a proposed change has been accepted or merged.
- Apache Iceberg — Signed-zero predicate evaluation, Issue #18723
- Apache Iceberg — Signed-zero predicate evaluation, PR #18813
- Apache Iceberg — JdbcLockFactory initialization, Issue #18797
- Apache Iceberg — JdbcLockFactory initialization, PR #18823
- Apache Iceberg — Parquet dictionary filtering and signed zero, Issue #18886
- Apache Spark — Floating-point row ordering, SPARK-60027
- Apache Spark — Development tooling, SPARK-60865
- Apache Airflow — Asynchronous S3Hook debug output, Issue #76852
- dbt Hub — GitHub Actions checkout compatibility, PR #4841
- dbt-jsonschema — Jinja Boolean configuration tests, PR #267
About the Interviewee
Aditya Ranjan is a Senior Data Engineer with more than 16 years of experience in enterprise data engineering, distributed systems, cloud data platforms, and AI infrastructure.
His independent public work includes technical issue investigations, proposed patches, and regression tests involving Apache Iceberg, Apache Spark, Apache Airflow, and dbt-related projects.
His writing and independent research focus on distributed-system correctness, storage efficiency, autonomous AI cost measurement, and governance-as-code for auditable AI operations.
Professional Profile: Aditya Ranjan — LinkedIn
Related DATAAD Article
Editorial Disclosure
This written interview was conducted, edited, and prepared for publication by JB Kim, Editor of DATAAD.COM. DATAAD independently developed the interview questions, selected the discussion topics, and prepared the editorial introduction.
Aditya Ranjan provided his responses and supporting public technical references in his personal capacity. His views are his own and do not represent those of his employer.
The interview has been formatted and section headings have been added for editorial clarity while preserving the substance of the interviewee's responses. The inclusion of external technical references does not imply endorsement by the referenced open-source projects or organizations.