Skip to content
Browse

/flags/pandas/read-csv-low-memory

pandas read_csv low_memory and DtypeWarning: Columns have mixed types

verified onenvironmentlocal · windows 10.0.26200 arm64 · GNU bash 5.3.15 (Git Bash)updated 2026-10-20
tool
pandas
command
pandas.read_csv
flag
low_memory, dtype
default
low_memory=True

pd.read_csv with the default low_memory=True parses large files in chunks and infers column types chunk by chunk. If one chunk of a column looks numeric and another contains text, pandas warns DtypeWarning: Columns (…) have mixed types and stores the column as object. Declare the type with dtype= (best), or pass low_memory=False so the whole column is inferred at once.

Reproduce the warning

A 300,001-line file where only the last value of val is not a number:

bash
python -c '
import warnings, pandas as pd
warnings.simplefilter("always")
with open("mixed.csv", "w") as f:
    f.write("id,val\n" + "\n".join(f"{i},1" for i in range(300000)) + "\n300000,x\n")
df = pd.read_csv("mixed.csv")
print("default:", df["val"].dtype)' 2>&1 | sed 's|^.*<string>|<string>|'

Output

<string>:6: DtypeWarning: Columns (1: val) have mixed types. Specify dtype option on import or set low_memory=False.
default: object

Fix: declare the type

bash
python -c '
import pandas as pd
with open("mixed.csv", "w") as f:
    f.write("id,val\n" + "\n".join(f"{i},1" for i in range(300000)) + "\n300000,x\n")
print("dtype=str:        ", pd.read_csv("mixed.csv", dtype={"val": "str"})["val"].dtype)
print("low_memory=False: ", pd.read_csv("mixed.csv", low_memory=False)["val"].dtype)
s = pd.to_numeric(pd.read_csv("mixed.csv", dtype={"val": "str"})["val"], errors="coerce")
print("numbers, bad -> NaN:", s.dtype, int(s.isna().sum()), "NaN")'

Output

dtype=str:         str
low_memory=False:  str
numbers, bad -> NaN: float64 1 NaN

dtype= documents your intent and avoids the second pass. When the column should be numeric, read it as text and convert with pd.to_numeric(..., errors="coerce") to find the bad rows.

OptionEffect
dtype={"col": "str"}force a type for one column (or a dict for several)
low_memory=Falseinfer each column from all its values; uses more memory
usecols=[...]read only the columns you need
engine="pyarrow"multithreaded parser when PyArrow is installed
na_values=["-", "n/a"]extra strings to treat as missing

Sources

Related pages

cheat sheets · pandaspandas cheat sheet: read, select, filter, group and reshapepandas cheat sheet: read CSV and Parquet, select with loc and iloc, filter rows, add columns, group and aggregate, pivot, handle missing values and write files.changes · pandaspandas 2 to 3: Copy-on-Write, the str dtype and other breaking changespandas 3.0 makes Copy-on-Write the only mode, infers strings as the new str dtype, drops SettingWithCopyWarning and needs Python 3.11+. Shown side by side.errors · numpyValueError: operands could not be broadcast together with shapesNumPy raises operands could not be broadcast together when array shapes are incompatible. Compare shapes from the right: each pair must be equal or 1.