03-Find-And-Delete-Duplicates_V01.qmd

Chapter-1-File-Organization-And-Backup

Author

Yahya Nazer - ChatBizDB.Com

Published

2026.06.18

1. Purpose

The Automated Duplicate File Mover is a comprehensive file management utility designed to process CSV reports containing duplicate file analysis and automatically move identified duplicate files to a secure staging area. This script provides an automated yet safe approach to duplicate file management, allowing users to efficiently clean up file systems while maintaining complete control over the deletion process.

Key Features:

  • Automated duplicate processing based on CSV report analysis
  • Safe file staging with move-first, delete-later approach
  • Cross-platform compatibility (Windows, macOS, Linux)
  • Intelligent CSV column detection with flexible mapping
  • Comprehensive conflict resolution with automatic filename suffixes
  • Detailed operation logging and audit trail generation
  • GUI-based file selection for user-friendly operation
  • Rollback capability through detailed move reports
  • UTF-8 encoding support for international filenames

Workflow Integration:

This script is designed to work seamlessly as part of a comprehensive file management workflow: 1. File Discovery: Use File Listing Script to generate comprehensive file inventories 2. Duplicate Analysis: Process listings to identify and mark duplicate files 3. Safe Staging: Use this script to move duplicates to review area 4. Manual Review: Verify duplicates before permanent deletion 5. Final Cleanup: Delete staging folder after confirmation

2. Inputs

System Requirements:

  • Python 3.6+ installed on your system
  • tkinter (usually included with Python installation)
  • Standard library modules only (no additional installations required)
  • File system permissions to create folders and move files

Required Input File:

The script requires one primary input:

CSV Duplicate Report File

A CSV file containing duplicate file analysis with specific column structure:

Column Position Expected Header Alternative Names Description Required Values
A (1st) Selector Any first column File classification “DUPLICATE” for files to move
C (3rd) Path File Path, FilePath Complete file system path Valid accessible file paths
I (9th) Duplicate_Index Index, Count Duplicate occurrence count Integer > 1 for files to move

Processing Logic:

Files are moved when both conditions are simultaneously met: - Column A (Selector) contains exactly “DUPLICATE” - Column I (Duplicate_Index) contains integer value > 1

Intelligent Column Detection:

The script features flexible column mapping:

Primary Detection (by header name):

  • Selector: Searches for “selector” keyword (case-insensitive)
  • Path: Searches for “path” keyword (case-insensitive)
  • Duplicate_Index: Searches for “duplicate_index” keyword or exact “I”

Fallback Detection (by position):

  • Selector: Uses first column (position A)
  • Path: Uses third column (position C)
  • Duplicate_Index: Uses ninth column (position I)

CSV Format Flexibility:

  • Delimiter auto-detection: Supports comma, tab, semicolon, and other separators
  • Encoding support: UTF-8 encoding for international characters
  • Header variations: Handles different naming conventions and case variations

3. Steps

Step 1: Script Execution

python 03-find-and-delete-duplicates-v01.py

Step 2: CSV File Selection

  • GUI Dialog: A file selection dialog opens automatically
  • File Navigation: Browse to your CSV duplicate report file
  • Format Validation: Select CSV files (.csv extension)
  • Confirmation: Click “Open” to proceed with selected file

Step 3: Automatic Staging Area Creation

The script automatically: - Folder Creation: Creates to_be_deleted folder in script directory - Permission Setup: Ensures proper write permissions - Path Validation: Confirms staging area accessibility - Status Reporting: Displays creation confirmation

Step 4: CSV Analysis and Processing

The script performs: - Structure Analysis: Reads and analyzes CSV column structure - Column Mapping: Maps columns to expected data types - Criteria Application: Applies filtering logic (DUPLICATE + Index > 1) - Progress Reporting: Displays real-time processing status

Step 5: File Moving Operations

For each qualifying file: - Existence Verification: Checks if file exists at specified path - Conflict Detection: Identifies potential filename conflicts - Safe Moving: Moves files to staging area with error handling - Naming Resolution: Applies numeric suffixes for conflicts - Operation Logging: Records all move operations

Step 6: Results Compilation and Reporting

  • Statistics Generation: Compiles operation statistics
  • Report Creation: Generates detailed move report
  • Console Summary: Displays operation results
  • Next Steps Guidance: Provides post-processing instructions

4. Outputs

Primary Staging Folder:

Location: ./to_be_deleted/ (created in script directory)

Contents: - All moved duplicate files with preserved original names - Conflict-resolved files with numeric suffixes (e.g., document_1.pdf, document_2.pdf) - Organized in flat structure for easy review

Naming Convention: - Original names preserved when no conflicts exist - Numeric suffixes applied for conflicts: filename_N.extension - Extension preservation maintains file type associations

Comprehensive Move Report:

File: ./move_report.txt (saved in script directory)

Report Structure and Contents:

Duplicate File Move Report
==================================================

Total files moved: [number]
Files not found: [number]  
Files with errors: [number]

SUCCESSFULLY MOVED FILES:
------------------------------
From: [original_file_path]
To:   [destination_path_in_staging]

[Repeated for each successfully moved file]

FILES NOT FOUND:
--------------------
[path_of_missing_file_1]
[path_of_missing_file_2]
[Additional missing files...]

FILES WITH ERRORS:
--------------------
File: [problematic_file_path]
Error: [detailed_error_description]

[Repeated for each error encountered]

Real-Time Console Output:

Processing Feedback: - Script initialization and version information - CSV file selection confirmation - Column detection results and mappings - Row-by-row processing status - File move confirmations or error messages - Operation summary with statistics

Example Console Output:

Duplicate File Mover
==================================================
Step 1: Select CSV file...
Selected CSV file: /path/to/duplicates.csv

Step 2: Creating 'to_be_deleted' folder...
Created folder: /script/directory/to_be_deleted

Step 3: Reading CSV and moving files...
CSV columns found: ['Selector', 'Filename', 'Path', ...]
Using columns: Selector='Selector', Duplicate_Index='I', Path='Path'

Processing row 15: /Users/documents/file1.pdf
  Moved to: /script/directory/to_be_deleted/file1.pdf
Processing row 23: /Users/documents/file2.pdf
  Moved to: /script/directory/to_be_deleted/file2_1.pdf

Processed 150 rows total

==================================================
OPERATION COMPLETE
==================================================
Files successfully moved: 12
Files not found: 2
Files with errors: 0

Operation Statistics:

The script tracks and reports: - Successfully moved files: Count and detailed paths - Missing files: Files referenced in CSV but not found on disk - Error files: Files that couldn’t be moved due to permissions or other issues - Processing summary: Total rows processed vs. files moved

Advanced Usage

Integration with File Listing Workflow:

  1. Generate File Inventory: Use 01-list-files-in-folder-v02.py to create comprehensive file listings
  2. Analyze for Duplicates: Process CSV to identify and mark duplicate files
  3. Stage Duplicates: Use this script to move duplicates to review area
  4. Manual Verification: Review staged files before final deletion
  5. Cleanup: Delete staging folder after verification

Batch Processing Considerations:

  • Large datasets: Script handles large CSV files efficiently
  • Memory usage: Processes files individually to minimize memory footprint
  • Error isolation: Individual file errors don’t stop batch processing
  • Progress tracking: Real-time feedback for long-running operations

Custom Column Mapping:

If your CSV uses different column names: 1. Header-based detection: Script searches for keywords in headers 2. Position-based fallback: Uses standard column positions as backup 3. Manual verification: Console output shows detected column mappings 4. Error reporting: Clear messages if required columns aren’t found

Safety and Recovery

Data Protection Features:

  • Non-destructive operations: Files are moved, never deleted automatically
  • Staging area isolation: All moved files accessible in single location
  • Detailed audit trail: Complete record of all move operations
  • Conflict resolution: Automatic filename suffixes prevent overwrites
  • Error handling: Graceful handling of permission and access issues

Recovery Procedures:

To restore moved files: 1. Locate staging folder: Navigate to to_be_deleted directory 2. Reference move report: Use move_report.txt for original file locations 3. Manual restoration: Move files back to original locations as needed 4. Batch restoration: Use move report as reference for scripted restoration

Emergency recovery: - All original file paths preserved in move report - Files remain accessible and intact in staging area - No data loss occurs during move operations

Troubleshooting

Common Issues and Solutions:

CSV Format Problems:

  • Issue: Column detection failures or unexpected CSV structure
  • Solution: Verify CSV has expected columns, check for proper encoding (UTF-8)
  • Debugging: Review console output for column mapping results

File Permission Errors:

  • Issue: Cannot move files or create staging folder
  • Solution: Run script with appropriate permissions, check file/folder ownership
  • Prevention: Ensure write access to script directory and source file locations

Missing Files:

  • Issue: Files referenced in CSV no longer exist
  • Solution: Check if files were previously moved or deleted, verify file paths
  • Impact: Missing files are logged but don’t stop processing of other files

Path Issues:

  • Issue: Incorrect or malformed file paths in CSV
  • Solution: Verify file paths use correct OS-specific format, check for special characters
  • Detection: Script reports path validation errors in console output

Performance Optimization:

  • Large file sets: Process during off-peak hours
  • Network drives: Consider local processing for network-stored files
  • Disk space: Ensure adequate space in staging area for moved files
  • Memory: Script is memory-efficient, suitable for large datasets

Best Practices

Pre-Processing Checklist:

  1. Backup critical data before running duplicate removal operations
  2. Verify CSV format matches expected column structure
  3. Check disk space for staging area requirements
  4. Test with small dataset before processing large file collections
  5. Review file paths for accuracy and accessibility

Post-Processing Workflow:

  1. Review staging area: Examine moved files to confirm they are duplicates
  2. Verify move report: Check for any errors or unexpected results
  3. Spot-check files: Open a few files to confirm they are indeed duplicates
  4. Gradual deletion: Consider deleting files in batches rather than all at once
  5. Final cleanup: Remove staging folder only after complete verification

Integration Tips:

  • Use consistent file naming conventions across workflow scripts
  • Maintain backup copies of original CSV reports
  • Document custom column mappings for future reference
  • Consider version control for script configurations

Resources & Downloads

Download all Python scripts and resources from this book:

www.chatbizdb.com/python-book

All code examples, templates, and additional materials are available for download. Please do not share this link - Thank you.

Copyright © 2025 ChatBizDB.com. All rights reserved.

No part of this publication may be reproduced, distributed, or transmitted in any form or by any means, including photocopying, recording, or other electronic or mechanical methods, without the prior written permission of the publisher, except in the case of brief quotations embodied in critical reviews and certain other noncommercial uses permitted by copyright law.

For permission requests, write to the publisher at: Email us

Contact Us

If you have any question or comment please contact us at Email us and make sure you add python-book at the beginning of the subject.

[1] "Time taken to run the Quarto document: 0.0911238193511963 seconds"