Duplicate Audit

Duplicate Audit scans local project folders, user directories, mounted storage, and network locations for possible duplicate files.

It can identify duplicate candidates using filenames, version-like naming patterns, and exact file-content hashes. Scanning is local and does not upload files or connect to external services.

A standard scan is non-destructive. Files are not moved or deleted unless the user explicitly selects files and executes the separate quarantine workflow. :contentReference[oaicite:2]{index=2}

Overview

Duplicate files can arise from:

• Manual copies

• Repeated downloads

• Backup folders

• Versioned filenames

• Data transfers

• Shared storage

• Renamed files with identical content

• Repeated exports from acquisition or analysis systems

Duplicate Audit helps users locate these candidates and prepare a reviewable cleanup plan.

Detection Modes

Same Filename

Detects files with the same filename in different directories.

This is a fast metadata-only comparison. It does not open file contents.

Files with the same name are not necessarily identical. They may represent different versions, participants, sessions, or processing outputs and must be reviewed manually.

Version-Like Files

Detects filenames containing patterns such as:

• copy

• old

• backup

• final

• final2

• v2

• version

• Numbered duplicate suffixes

This mode identifies likely duplicate or superseded files based on naming only.

Exact Content Hash

Detects files with exactly matching binary content, even when their filenames or locations differ.

NIM Studio first groups files by size and hashes only files that share a size with another candidate. This reduces unnecessary file reading.

Supported algorithms are:

• BLAKE3

• SHA-256

BLAKE3 is recommended for large storage scans when the optional blake3 package is installed. SHA-256 is available through Python.

Hash scanning opens and reads local file bytes but does not modify them. :contentReference[oaicite:3]{index=3}

Scan Scope

Standard Recursive Scan

The standard scan recursively searches the selected root.

Recommended targets include:

• A single research project

• A dataset root

• Documents

• Downloads

• Desktop

• A mounted project drive

• A specific institutional storage location

Deep System or Storage Scan

The deep scan is intended for broader storage locations and may take considerably longer.

Use this mode carefully because it may encounter:

• System folders

• Restricted files

• Cloud placeholders

• Symbolic links

• Large archives

• Network latency

• Permission errors

• Files owned by multiple users

Start with a narrowly defined project root whenever possible.

Safety Filters

Depending on the selected configuration, the audit can skip:

• Hidden files

• System directories

• Cloud placeholder files

• Symbolic links

• Unreadable files

• Files outside a selected owner filter

• Empty files during content hashing

• Files above a configured hash-size limit

Cloud placeholder skipping should remain enabled when scanning directories synchronized through cloud services. Otherwise, opening a placeholder for hashing may cause the cloud client to download it.

Owner Filtering

On shared systems, Duplicate Audit can attempt to identify file owners.

When the owner filter is enabled:

1. Select the audit root.

2. Click Discover owners.

3. Review the detected owner list.

4. Select the owner whose files should be included.

5. Run the duplicate scan.

Owner detection depends on the operating system and permissions. Ownership information may be unavailable or may use a fallback value. :contentReference[oaicite:4]{index=4}

How to Run a Duplicate Audit

1. Open the Duplicate Audit module.

2. Click Browse and select the folder to scan.

3. Choose the scan scope.

   Start with Standard recursive scan for normal project-level audits.

4. Keep the recommended safety filters enabled.

5. Select one or more detection modes:

   • Same filename

   • Version-like files

   • Exact same content fingerprint

6. When using exact-content detection, choose BLAKE3 or SHA-256.

7. Review the warning explaining that hash scanning reads local file contents.

8. Start the scan.

9. Monitor the progress indicators showing:

   • Files seen

   • Files retained after filtering

   • Candidate files

   • Files hashed

   • Duplicate groups

10. Review the candidate table after the scan completes.

11. Do not assume that the older file should always be removed.

12. Check the path, reason, size, modification time, owner, and hash before selecting any file.

The scan report explicitly states that nothing has been moved or deleted. :contentReference[oaicite:5]{index=5}

Reviewing Results

The result table includes:

• Older-file selection

• Newer-file selection

• Older path

• Newer path

• Recommended action

• File owner

• Detection reason

• Duplicate group

• File size

• Modification time

• Content hash

By default, the older file may be preselected while the newer file remains unselected. This is only a review aid and is not proof that the older file is unnecessary. :contentReference[oaicite:6]{index=6}

Before selecting a candidate, consider:

• Is one file located in an official raw-data location?

• Is one file part of a derivative dataset?

• Is one file linked from code or documentation?

• Do the files have the same content hash?

• Is one file the only protected backup?

• Are both files required for provenance?

• Is the newer timestamp actually meaningful?

• Does one filename indicate an approved final version?

• Are the files owned by different users?

Exporting Reports

Duplicate Audit can export selected results as CSV.

The exported rows may include:

• Selected older and newer files

• File paths

• Recommended action

• User-selected action

• Owner

• Detection reason

• Duplicate group

• Size

• Modification time

• Content hash

The full audit package can also include:

• All duplicate candidates

• Selected candidates

• Owner summary

• Scan diagnostics

• Quarantine manifest preview

The report export does not move any files. :contentReference[oaicite:7]{index=7}

Quarantine Preview

Before moving any files, use the quarantine preview.

The preview shows:

• Selected candidate files

• Proposed quarantine locations

• Detection reasons

• Number of files selected

Previewing the quarantine manifest does not move files. :contentReference[oaicite:8]{index=8}

Preview-Only Mode

When Preview only is enabled:

• No files are moved.

• No files are deleted.

• A quarantine plan can still be reviewed and exported.

• Selected files remain in their original locations.

Preview-only mode is recommended during beta testing.

Quarantine

Quarantine is a separate, explicit operation.

When executed, selected files are moved into a quarantine directory rather than deleted. NIM Studio requests confirmation before the move.

If a file with the same name already exists in quarantine, NIM Studio adds a duplicate suffix rather than silently replacing it.

The quarantine process may generate a manifest recording:

• Original file path

• Quarantine destination

• Detection reason

• File size

• Hash

• Move status

• Timestamp

Quarantined files should be retained until the dataset or project has been checked and all workflows continue to operate correctly. :contentReference[oaicite:9]{index=9}

Recommended Safe Workflow

1. Create or verify a current backup.

2. Select a narrow audit root.

3. Run filename and version-like detection first.

4. Review the initial candidate set.

5. Enable exact-content hashing for candidates that require stronger evidence.

6. Export the complete audit package.

7. Review candidate paths with the project owner or data steward.

8. Use the quarantine preview.

9. Keep preview-only mode enabled during initial testing.

10. Select files individually rather than bulk-selecting all older files.

11. Export the quarantine manifest.

12. Move approved candidates to quarantine only after review.

13. Re-run analyses, scripts, or project checks.

14. Retain the quarantine directory for an agreed review period.

15. Delete quarantined files manually only after independent confirmation.

What Duplicate Audit Does Not Guarantee

Duplicate Audit does not determine that a file is safe to remove.

In particular:

• Matching filenames do not prove matching content.

• Version-like names do not prove that a file is obsolete.

• Identical hashes prove identical file bytes, but not that both locations are unnecessary.

• Modification time does not prove which copy is authoritative.

• The older file is not automatically the less important file.

• Duplicate files may be intentional backups.

• Duplicate files may be required by separate projects.

• Symbolic links, permissions, cloud placeholders, and network shares can affect scan results.

• Ownership information may be incomplete.

• The audit does not understand every scientific dependency.

• The audit does not inspect code to determine whether a file is referenced.

• The audit does not delete files automatically.

Beta Safety Notice

During the beta phase:

• Use test data or a backed-up project.

• Keep preview-only mode enabled initially.

• Avoid scanning an entire operating system drive.

• Avoid moving source data before checking project dependencies.

• Export reports before quarantine.

• Review every selected file manually.

• Treat quarantine as reversible temporary storage, not immediate deletion.