Eddie Atkinson’s argument is not that Pandas is unpopular — it’s that Pandas’ performance cliff chooses your architecture for you. Teams hit memory pressure in the tens of gigabytes and reach for Spark, Databricks, Snowflake or Dask: distributed systems they will probably never need. He gave the talk at Latency Conference and published the benchmarks with it.

How much “Big Data” is actually out there:

  • Amazon’s own analysis of its Redshift fleet puts 94.68% of tables under 100GB
  • 86.9% of queries operate on 80GB or less, and finish in under a second
  • Even assuming 10KB rows, the ceiling is 1TB — “You have Medium Data problems, and need Medium Data solutions”

On the 1 Billion Row Challenge, 32 cores and 128GB:

  • Pandas: 4m 28s, pinned at 113% CPU, 38GB peak memory
  • Polars: 5.04s, 3202% CPU, 18GB
  • DuckDB: 5.19s, 3174% CPU, 1.9GB

That is roughly 53 times faster, and DuckDB got there with no tuning and about a twentieth of the memory. On a Framework 13 laptop the gap gets uglier: Pandas took 12m 15s and burned 21GB of swap, while DuckDB took 47s. That last one is the bill most teams actually pay — a slow inner dev loop, not a cluster invoice.

The mechanics explain the numbers. Polars scans lazily and builds an optimized query graph before executing; DuckDB runs SQL over your data wherever it lives. Both stream in chunks, parallelize across cores, push predicates down before aggregating, and spill to disk instead of leaning on the OS. Pandas reads everything into memory first and then works through it eagerly.

Switching does not require a rewrite. Apache Arrow — the columnar format from Pandas’ own creator, Wes McKinney — is native to Polars and DuckDB and supported by Pandas since 2.0, so frames move between them without copying memory. The catch is that Pandas will not use Arrow backing unless you ask for it with dtype_backend="pyarrow". His mixed benchmarks show either half can be swapped independently: pure Pandas took 41.88s on 3GB of taxi parquet, while DuckDB reading and Pandas computing took 28.39s.

Two things make the post worth trusting. The benchmark code is open source, and he publishes his own counterarguments — including that a couple of the third-party benchmarks he cites come from vendors competing with Polars and DuckDB. The takeaway is narrow: the only thing you shouldn’t do is adopt a distributed query engine because your local tool is slow.