Pandas often has significant memory overhead, so it's not uncommon to need ~3-5x... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		datadrivenangel on May 8, 2024 \| parent \| context \| favorite \| on: The Birth of Parquet Pandas often has significant memory overhead, so it's not uncommon to need ~3-5x the amount of memory as your file size. Polars and DuckDB are much better about memory management.

marton78 on May 8, 2024 [–]

Also, Polars has a sane and well thought out API, unlike the hot streaming mess that is Pandas.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact