DupFileManager

[David-Maisonave]

	Summary	Manages duplicate files.
	Repository	https://github.com/stashapp/CommunityScripts/blob/main/plugins/DupFileManager
	Source URL	https://stashapp.github.io/CommunityScripts/stable/index.yml
	Install	How to install a plugin?

DupFileManager is a Stash plugin which manages duplicate files in the Stash system.
It has both task and tools-UI components.

Features

• Creates a duplicate file report which can be accessed from the settings->tools menu options.The report is created as an HTML file and stored in local path under plugins\DupFileManager\report\DuplicateTagScenes.html.
  • See screenshot at the bottom of this page for example report.
  • Items on the left side of the report are the primary duplicates designated for deletion. By default, these duplicates are given a special _duplicate tag.
  • Items on the right side of the report are designated as primary duplicates to keep. They usually have higher resolution, duration and/or preferred paths.
  • The report has the following options:
    • Delete: Delete file and remove from Stash library.
    • Remove: Remove from Stash library.
    • Rename: Rename file.
    • Copy: Copy file from left (source) to right (to-keep).
    • Move: Copy file and metadata left to right.
    • Cpy-Name: Copy file name left to right.
    • Add-Exclude: Add exclude tag to scene,so that scene is excluded from deletion.
    • Remove-Tag: Remove duplicate tag from scene.
    • Flag-Scene: Flag (mark) scene in report as reviewed (or as requiring further review). Optional flags (yellow, green, orange, cyan, pink, red, strike-through, & disable-scene)
    • Merge: Copy Metadata (tags, performers,& studios) from left to right.
• Can merge potential source in the duplicate file names for tag names, performers, and studios.
  • Normally when Stash searches the file name for tag names, performers, and studios, it only does so using the primary file.
• Advance menu (for specially tagged duplicates)
  
  
  • Advance menu can be access from the Settings->Tools->[DupFileManager Tools and Utilities] menu or from the reports.
    • Only access Advance Menu from the report when using Stash setup requiring a password.
  • Here are some of the options available in the Advance Menu.
    • Delete only specially tagged duplicates in blacklist path.
    • Delete duplicates with specified file path.
    • Delete duplicates with specific string in File name.
    • Delete duplicates with specified file size range.
    • Delete with specified duration range.
    • Delete with resolution range.
    • Delete duplicates having specified tags.
    • Delete duplicates with specified rating.
    • Delete duplicates with any of the above combinations.
• Bottom extended portion of the Advanced Menu screen.
  • 
• Delete duplicate file task with the following options:
  • Tasks (Settings->Task->[Plugin Tasks]->DupFileManager)
    • Tag Duplicates - Set tag DuplicateMarkForDeletion to the duplicates with lower resolution, duration, file name length, and/or black list path.
    • Delete Tagged Duplicates - Delete scenes having DuplicateMarkForDeletion tag.
    • Delete Duplicates - Deletes duplicate files. Performs deletion without first tagging.
  • Plugin UI options (Settings->Plugins->Plugins->[DupFileManager])
    • Has a 3 tier path selection to determine which duplicates to keep, and which should be candidates for deletions.
      • Whitelist - List of paths NOT to be deleted.
        • E.g. C:\Favorite,E:\MustKeep\
      • Gray-List - List of preferential paths to determine which duplicate should be the primary.
        • E.g. C:\2nd_Favorite,H:\ShouldKeep\
      • Blacklist - List of LEAST preferential paths to determine primary candidates for deletion.
        • E.g. C:\Downloads,F:\DeleteMeFirst\
    • Permanent Delete - Enable to permanently delete files, instead of moving files to trash can.
    • Max Dup Process - Use to limit the maximum files to process. Can be used to do a limited test run.
    • Merge Duplicate Tags - Before deletion, merge metadata from duplicate. E.g. Tag names, performers, studios, title, galleries, rating, details, etc…
    • Swap High Resolution - When enabled, swaps higher resolution files between whitelist and blacklist/graylist files.
    • Swap Longer Duration - When enabled, swaps scene with longer duration.
  • Options available via DupFileManager_config.py
    • dup_path - Alternate path to move deleted files to. Example: “C:\TempDeleteFolder”
    • toRecycleBeforeSwap - When enabled, moves destination file to recycle bin before swapping files.
    • addPrimaryDupPathToDetails - If enabled, adds the primary duplicate path to the scene detail.
• Tools UI Menu
  
  
  • Can access either Duplicate File Report (DupFileManager) or DupFileManager Tools and Utilities menu options.

Requirements

• pip install --upgrade stashapp-tools
• pip install requests
• pip install Send2Trash

Installation

• Follow Requirements instructions.
• In the stash plugin directory (C:\Users\MyUserName.stash\plugins), create a folder named DupFileManager.
• Copy all the plugin files to this folder.(C:\Users\MyUserName\.stash\plugins\DupFileManager).
• Click the [Reload Plugins] button in Stash->Settings->Plugins->Plugins.

That’s it!!!

Options

• Options are accessible in the GUI via Settings->Plugins->Plugins->[DupFileManager].
• More options available in DupFileManager_config.py.

Screenshots

• Example DupFileManager duplicate report. (file names have been edited to PG).
  • The report displays preview videos that are playable. Will play a few seconds sample of the video. This requires scan setting [Generate animated image previews] to be enabled when scanning all files.
  • 
  • If there’s a scene on the left side that has a higher resolution or duration, it gets a yellow highlight on the report.
  • There’s an optional setting that allows both preview videos and preview images to be displayed on the report. See settings htmlIncludeImagePreview in the DupFileManager_report_config.py file.
  • There are many more options available for how the report is created. These options are targeted for more advanced users. The options are all available in the DupFileManager_report_config.py file, and the settings have commented descriptions preceeding them. See the DupFileManager_report_config.py file in the DupFileManager plugin folder for more details.
• Tools UI Menu
  
  
  • Can access either Duplicate File Report (DupFileManager) or DupFileManager Tools and Utilities menu options.
• DupFileManager Report Menu
  • 
• DupFileManager Tools and Utilities
  • 
• Full bottom extended portion of the Advanced Menu screen.
  • 

Future Planned Features

• Add logic to merge performers and galaries seperatly from tag merging on report. Planned for 1.5.0 Version.
• Add code to report to make it when the report updates the screen (due to tag merging), it stays in the same row position. Planned for 1.5.0 Version.
• Add logic to merge group metadata when selecting merge option on report. Planned for 2.0.0 Version.
• Add advanced menu directly to the Settings->Tools menu. Planned for 2.0.0 Version.
• Add report directly to the Settings->Tools menu. Planned for 2.0.0 Version.

[KlutzyBubbles]

Has anyone been able to get this working? I am on windows and been trying to see if this can help me merge like 1000 duplicates but I can’t load the report even though the report file exists.

Does anyone know of any way to bulk merge duplicates? I have transcoded a bunch of files from mp4 to mkv but want to keep all the tags from the mp4s onto the mkv but the only way i have found to do this so far is one at a time in sceneDuplicateChecker

[Hellfire702]

Don’t know if this is the right place to put this, but upon following the install instructions & attempting to run either tool from this plugin, it leads Stash to crash with the following error:

TypeError: AjaxData.responseJSON.data.runPluginOperation is null
    ToolsAndUtilities@http://localhost:9999/plugin/DupFileManager/javascript:244:9
    t@http://localhost:9999/assets/index-31134d58.js:40:9217
    apply@http://localhost:9999/assets/index-31134d58.js:44:185328
    t@http://localhost:9999/assets/index-31134d58.js:40:11169
    Suspense
    MY@http://localhost:9999/assets/index-31134d58.js:44:448755
    div
    jy2@http://localhost:9999/assets/index-31134d58.js:54:54934
    qg2@http://localhost:9999/assets/index-31134d58.js:49:4022
    C52@http://localhost:9999/assets/index-31134d58.js:44:188954
    Suspense
    M52@http://localhost:9999/assets/index-31134d58.js:44:186944
    Ug2@http://localhost:9999/assets/index-31134d58.js:49:3803
    apply@http://localhost:9999/assets/index-31134d58.js:44:185368
    TD2@http://localhost:9999/assets/index-31134d58.js:64:32661
    t@http://localhost:9999/assets/index-31134d58.js:40:75570
    MY@http://localhost:9999/assets/index-31134d58.js:44:448755
    c$2@http://localhost:9999/assets/index-31134d58.js:202:17903
    XZ@http://localhost:9999/assets/index-31134d58.js:19:183839
    t@http://localhost:9999/assets/index-31134d58.js:40:5733
    t@http://localhost:9999/assets/index-31134d58.js:40:12243

[LudeDevil]

Hi there! DupFileManager looks like a fantastic addition to Stash. I am trying really hard to get it installed and working in my Docker image. I had to manually create the ‘reports’ directory, as the python code is looking in the wrong path. Once that was done, I get a new error (or warning) when running the report generation (No Tag). The error is:

[Plugin / DupFileManager] [LN:517] WRN: 'Exception calling [mergeMetadata]. Will retry; count(0); Error: \'id\'\nTraceBack=Traceback (most recent call last):\n  File "/config/plugins/community/DupFileManager/StashPluginHelper.py", line 521, in mergeMetadata\n    return self._mergeMetadata.merge(SrcData, DestData)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/config/plugins/community/DupFileManager/StashPluginHelper.py", line 996, in merge\n    self.mergeItems(\'performers\', \'performer_ids\', [])\n  File "/config/plugins/community/DupFileManager/StashPluginHelper.py", line 1045, in mergeItems\n    listToAdd += [item[\'id\']]\n                  ~~~~^^^^^^\nKeyError: \'id\'\n'

Does this give you any clue as to what is going wrong?

[David-Maisonave]

The author of this plugin does not monitor Discourse. Please post any bugs, feature request, help-request to the following link: https://github.com/David-Maisonave/Axter-Stash/issues/new/choose.
The Discourse link should be use for discussion that would be inappropriate in GitHub.