DupFileManager

[DogmaDragon]

Notice
If you are the author of this plugin, you can claim this topic by clicking > at the bottom of this post. Select Something Else as a reason and leave a comment explaining you want to claim the topic. Include a link to your git profile or pull request showing ownership of the plugin.

	Summary	Manages duplicate files.
	Repository	https://github.com/stashapp/CommunityScripts/blob/main/plugins/DupFileManager
	Source URL	https://stashapp.github.io/CommunityScripts/stable/index.yml
	Install	How to install a plugin?

DupFileManager is a Stash plugin which manages duplicate files in the Stash system.
It has both task and tools-UI components.

Features

• Creates a duplicate file report which can be accessed from the settings->tools menu options.The report is created as an HTML file and stored in local path under plugins\DupFileManager\report\DuplicateTagScenes.html.
  • See screenshot at the bottom of this page for example report.
  • Items on the left side of the report are the primary duplicates designated for deletion. By default, these duplicates are given a special _duplicate tag.
  • Items on the right side of the report are designated as primary duplicates to keep. They usually have higher resolution, duration and/or preferred paths.
  • The report has the following options:
    • Delete: Delete file and remove from Stash library.
    • Remove: Remove from Stash library.
    • Rename: Rename file.
    • Copy: Copy file from left (source) to right (to-keep).
    • Move: Copy file and metadata left to right.
    • Cpy-Name: Copy file name left to right.
    • Add-Exclude: Add exclude tag to scene,so that scene is excluded from deletion.
    • Remove-Tag: Remove duplicate tag from scene.
    • Flag-Scene: Flag (mark) scene in report as reviewed (or as requiring further review). Optional flags (yellow, green, orange, cyan, pink, red, strike-through, & disable-scene)
    • Merge: Copy Metadata (tags, performers,& studios) from left to right.
• Can merge potential source in the duplicate file names for tag names, performers, and studios.
  • Normally when Stash searches the file name for tag names, performers, and studios, it only does so using the primary file.
• Advance menu (for specially tagged duplicates)
  
  
  • Advance menu can be access from the Settings->Tools->[DupFileManager Tools and Utilities] menu or from the reports.
    • Only access Advance Menu from the report when using Stash setup requiring a password.
  • Here are some of the options available in the Advance Menu.
    • Delete only specially tagged duplicates in blacklist path.
    • Delete duplicates with specified file path.
    • Delete duplicates with specific string in File name.
    • Delete duplicates with specified file size range.
    • Delete with specified duration range.
    • Delete with resolution range.
    • Delete duplicates having specified tags.
    • Delete duplicates with specified rating.
    • Delete duplicates with any of the above combinations.
• Bottom extended portion of the Advanced Menu screen.
  • 
• Delete duplicate file task with the following options:
  • Tasks (Settings->Task->[Plugin Tasks]->DupFileManager)
    • Tag Duplicates - Set tag DuplicateMarkForDeletion to the duplicates with lower resolution, duration, file name length, and/or black list path.
    • Delete Tagged Duplicates - Delete scenes having DuplicateMarkForDeletion tag.
    • Delete Duplicates - Deletes duplicate files. Performs deletion without first tagging.
  • Plugin UI options (Settings->Plugins->Plugins->[DupFileManager])
    • Has a 3 tier path selection to determine which duplicates to keep, and which should be candidates for deletions.
      • Whitelist - List of paths NOT to be deleted.
        • E.g. C:\Favorite,E:\MustKeep\
      • Gray-List - List of preferential paths to determine which duplicate should be the primary.
        • E.g. C:\2nd_Favorite,H:\ShouldKeep\
      • Blacklist - List of LEAST preferential paths to determine primary candidates for deletion.
        • E.g. C:\Downloads,F:\DeleteMeFirst\
    • Permanent Delete - Enable to permanently delete files, instead of moving files to trash can.
    • Max Dup Process - Use to limit the maximum files to process. Can be used to do a limited test run.
    • Merge Duplicate Tags - Before deletion, merge metadata from duplicate. E.g. Tag names, performers, studios, title, galleries, rating, details, etc…
    • Swap High Resolution - When enabled, swaps higher resolution files between whitelist and blacklist/graylist files.
    • Swap Longer Duration - When enabled, swaps scene with longer duration.
  • Options available via DupFileManager_config.py
    • dup_path - Alternate path to move deleted files to. Example: “C:\TempDeleteFolder”
    • toRecycleBeforeSwap - When enabled, moves destination file to recycle bin before swapping files.
    • addPrimaryDupPathToDetails - If enabled, adds the primary duplicate path to the scene detail.
• Tools UI Menu
  
  
  • Can access either Duplicate File Report (DupFileManager) or DupFileManager Tools and Utilities menu options.

Requirements

• pip install --upgrade stashapp-tools
• pip install requests
• pip install Send2Trash

Installation

• Follow Requirements instructions.
• In the stash plugin directory (C:\Users\MyUserName.stash\plugins), create a folder named DupFileManager.
• Copy all the plugin files to this folder.(C:\Users\MyUserName\.stash\plugins\DupFileManager).
• Click the [Reload Plugins] button in Stash->Settings->Plugins->Plugins.

That’s it!!!

Options

• Options are accessible in the GUI via Settings->Plugins->Plugins->[DupFileManager].
• More options available in DupFileManager_config.py.

Screenshots

• Example DupFileManager duplicate report. (file names have been edited to PG).
  • The report displays preview videos that are playable. Will play a few seconds sample of the video. This requires scan setting [Generate animated image previews] to be enabled when scanning all files.
  • 
  • If there’s a scene on the left side that has a higher resolution or duration, it gets a yellow highlight on the report.
  • There’s an optional setting that allows both preview videos and preview images to be displayed on the report. See settings htmlIncludeImagePreview in the DupFileManager_report_config.py file.
  • There are many more options available for how the report is created. These options are targeted for more advanced users. The options are all available in the DupFileManager_report_config.py file, and the settings have commented descriptions preceeding them. See the DupFileManager_report_config.py file in the DupFileManager plugin folder for more details.
• Tools UI Menu
  
  
  • Can access either Duplicate File Report (DupFileManager) or DupFileManager Tools and Utilities menu options.
• DupFileManager Report Menu
  • 
• DupFileManager Tools and Utilities
  • 
• Full bottom extended portion of the Advanced Menu screen.
  • 

Future Planned Features

• Add logic to merge performers and galaries seperatly from tag merging on report. Planned for 1.5.0 Version.
• Add code to report to make it when the report updates the screen (due to tag merging), it stays in the same row position. Planned for 1.5.0 Version.
• Add logic to merge group metadata when selecting merge option on report. Planned for 2.0.0 Version.
• Add advanced menu directly to the Settings->Tools menu. Planned for 2.0.0 Version.
• Add report directly to the Settings->Tools menu. Planned for 2.0.0 Version.

[KlutzyBubbles]

Has anyone been able to get this working? I am on windows and been trying to see if this can help me merge like 1000 duplicates but I can’t load the report even though the report file exists.

Does anyone know of any way to bulk merge duplicates? I have transcoded a bunch of files from mp4 to mkv but want to keep all the tags from the mp4s onto the mkv but the only way i have found to do this so far is one at a time in sceneDuplicateChecker