pandas: Working with Tables
DataFrames: loading, inspecting, selecting, filtering, creating columns, grouping, merging, reshaping and dates.
Series and DataFrames
pandas gives Python spreadsheets that you control with code. A DataFrame is a table: named columns, each holding one type, and an index labelling the rows. A single column is a Series. Under the hood each column is a NumPy array, so vectorised operations are fast.
Always look before you model: head, info, describe and value_counts catch most data problems in the first five minutes (wrong types, impossible values, missing data, surprising categories).
Selecting, filtering and new columns
Avoid looping over rows (for i, row in df.iterrows()): it is slow and rarely needed. Express the rule as a column operation instead. When you filter and then want to change the result, add .copy() to avoid the SettingWithCopyWarning.
Try it: pandas operations, step by step
Pick an operation to see the pandas code and the exact table it produces from a small appointments table.
Interactive: this part runs in your browser.
Code example: pandas essentials
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Grouping, merging, reshaping and dates
groupby splits the table into groups, applies a function to each, and combines the results: the same idea as SQL's GROUP BY.
merge joins tables like SQL joins. Always check the row count: if the key is not unique on the right, rows multiply.
Reshaping: pivot_table turns long data into a grid (rows × columns); melt does the reverse. Dates: convert with pd.to_datetime, then use the .dt accessor (.dt.month, .dt.dayofweek), subtract dates to get time differences, and resample("W") to aggregate by week. Missing values: find them with isna().sum(), then decide per column whether to drop, fill (fillna) or flag them, as the Data Cleaning topic explains.
Code example: merge, pivot and dates
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Cheat sheet: pandas
The commands you will use most, on one screen. Come back to it while you work.
Interactive: this part runs in your browser.
Quiz
1. Which command shows column types and missing-value counts?
2. Which selects rows where clinic is North AND age > 60?
3. df.groupby("clinic")["no_show"].mean() returns…
4. A left merge increases the number of rows. Why?
5. Why avoid iterrows() loops?
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- 10 minutes to pandas ↗ Official pandas quick start
- Kaggle Learn ↗ Free short courses with notebooks