Handling Duplicates & Data Type Conversions0%
Applying Custom Functions (apply, map, applymap)

Handling Duplicates & Data Type Conversions

Beginner12 min read•Updated: Jul 11, 2026
Study MaterialsCheatsheet

Handling Duplicates & Data Type Conversions

Reading Time: 8 Mins | Level: Beginner to Advanced

1. Detecting & Removing Duplicate Records

Duplicate rows often occur due to repeated form submissions, database re-syncs, or ETL joining errors.

Python
import pandas as pd
 
df = pd.DataFrame({
'Transaction_ID': [101, 102, 103, 102, 104],
'User': ['Amit', 'Priya', 'Rohan', 'Priya', 'Sneha'],
'Amount': [1200, 4500, 3100, 4500, 8900]
})
 
# 1. Identify duplicates
print(df.duplicated()) # Returns boolean Series
 
# 2. Check for duplicate Transaction IDs only
print(df.duplicated(subset=['Transaction_ID']))
 
# 3. Drop duplicate rows (keep the first occurrence)
df_unique = df.drop_duplicates(subset=['Transaction_ID'], keep='first')
print(df_unique)

The keep Parameter Options:

  • keep='first' (Default): Retains the first occurrence and drops subsequent duplicates.
  • keep='last': Retains the last occurrence and drops earlier duplicates.
  • keep=False: Drops ALL occurrences of duplicates (keeping only strictly unique rows).

2. Converting Data Types (.astype())

Often numbers (like "₹4500" or phone numbers "9876543210") are imported as generic object (string) types. To perform calculations, cast them to proper datatypes:

Python
# Converting data types
df['Transaction_ID'] = df['Transaction_ID'].astype(str)
df['Amount'] = df['Amount'].astype(float)

Safe Conversion with pd.to_numeric()

When a column contains dirty strings (like "N/A" or "invalid"), .astype(float) crashes with a ValueError. pd.to_numeric(errors='coerce') gracefully converts bad values into NaN:

Python
raw_prices = pd.Series(['150.50', '200.00', 'corrupted_val', '310.25'])
 
# errors='coerce' turns 'corrupted_val' into NaN instead of crashing
clean_prices = pd.to_numeric(raw_prices, errors='coerce')
print(clean_prices)

3. The Categorical Data Type (category)

For text columns with limited unique values (e.g. Gender, State, Department), converting from object to category reduces memory usage by up to 90% and speeds up grouping:

Python
df['User'] = df['User'].astype('category')
Interactive Quiz

Multiple Choice Questions

Question 1
Which method drops duplicate records from a DataFrame?
Question 2
What happens when calling df.drop_duplicates(keep=False)?
Question 3
What does pd.to_numeric(series, errors='coerce') do when encountering unparseable text?
Question 4
Which datatype should be used for text columns with low cardinality (few unique strings repeated millions of times) to save memory?
Question 5
How do you convert an integer column 'Age' into a 32-bit float in Pandas?
Next Lesson

Applying Custom Functions (apply, map, applymap)

Continue learning with hands-on practice, examples, and exercises in the upcoming topic.

Related Lessons

Previous LessonNext Lesson
Dropping & Filling Missing Values (dropna, fillna)Applying Custom Functions (apply, map, applymap)

Practice Quiz

Test your understanding of this lesson with 5 questions. Each question has one correct answer.

PreviousNext
PrevNext