CSV Deduplication

How HostMetrics prevents duplicate records on re-import

CSV Deduplication

When users import CSVs (manually or via Chrome extension), the system prevents duplicate records.

Deduplication Keys

Data Type	Unique Key	Strategy
Trip Earnings	(user_id, reservation_id)	Upsert — update existing on conflict
Earnings Reports	(user_id, composite_key)	Upsert — update existing on conflict
Toll Transactions	(provider, date, amount, plaza)	Skip — ignore duplicates

Trip Earnings Deduplication

Each Turo trip has a unique reservation_id. When importing:

1. Check if a trip with the same (user_id, reservation_id) exists
2. If yes → update the existing record (trip details may change, e.g., status Completed → Paid)
3. If no → insert a new record
4. This is done in batch upserts of 500 rows

Import Superseding

When a new CSV is imported:

1. A new csv_imports audit record is created with status active
2. Previous imports of the same type are marked as superseded
3. The actual trip/earning records remain — they’re upserted, not replaced
4. This provides an audit trail of import history

Chrome Extension Sync

The Chrome extension follows the same deduplication logic:

1. Extension scrapes Turo page → sends CSV content
2. Server parses CSV identically to manual upload
3. Same upsert logic applies
4. Import record is tagged with chrome_extension in the filename
5. The Data Sources page shows “Last synced: 2h ago” from the import record

Re-Import Safety

It is safe to re-import the same CSV multiple times:

• Existing records are updated (not duplicated)
• New records are inserted
• Import count shows “X new, Y updated”
• No data loss or corruption occurs