Migration job not running on upload dir

[joojoooo]

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

• Yes

The bug

When media is uploaded, it's temporarily saved in the /upload/uploaddirectory. If moving it to the /upload/library fails (for whatever reason), the file is stuck forever in upload, the migration job won't try to move it to the storage template location.
I mount library on a network drive and this can sometimes happen, rarely, but it happens (network issues, power outage, etc...) . Uploads are still successful, just never moved to the library.

I think keeping files in the upload directory is confusing also for making backups, I never thought I had to backup upload too... IMHO Immich should try to move any files left in the upload dir regularly or on startup but if that's not possible we should at least be able to use the migration job to move them.

The OS that Immich Server is running on

Ubuntu 24.04.2 LTS

Version of Immich Server

v1.131.3

Platform with the issue

• Server
• Web
• Mobile

Reproduction steps

1. Make library immutable to simulate a move error: chattr -R +i upload/library/
2. Upload media to get this warning in logs: Unable to complete move. Error renaming file
3. Remove immutable: chattr -R -i upload/library/
4. Run migration job
5. Notice the file not moving from upload/upload

Relevant log output

Removed not to cause confusion on what's the actual issue

[mmomjian]

The error indicates the move is being attempted, but a permission issue on the network mount is preventing the move from being completed.

[joojoooo]

You didn't understood the issue. The permission error is caused on purpose to reproduce the issue. The issue has nothing to do with permissions but with real world unpredictable events. It's about media location and migration job not moving the file AFTER the move error is fixed. The log is about the error happening on upload due to force majeure, that can't be fixed, the issue is after the upload and after the error is taken care of

[joojoooo]

I don't think you even read the issue, you just took a quick look at the log 😭

[mmomjian]

I read the issue. If that’s the case, you didn’t share any logs from when you actually try to run the migration job, which we need.

We would also need to see the issue replicated on a local storage to confirm it’s not a network mount issue.

[mmomjian]

There will be a log line for “started” and “ended” storage template migration. This should be logged by default unless you have changed the log level.

Replicate locally = without network mount. It’s not clear to me that you already tested that?

Your post in general is very light on details. I’m not sure why you removed the compose and env sections, for example.

[joojoooo]

Sorry 😅, now it's working and I've no idea why, I had logs set to warning only

[Nest] 7  - 04/15/2025, 12:32:29 PM     LOG [Microservices:StorageTemplateService] Starting storage template migration

[Nest] 7  - 04/15/2025, 12:32:29 PM     LOG [Microservices:ApiKeyService] Attempting to finish incomplete move: upload/upload/2d4a1860-8892-45da-9036-64f29b012f2a/7a/4c/7a4c8a0a-8281-4a4e-ab3a-e6bddaefcd80.avif => upload/library/admin/2024/03/test.avif

[Nest] 7  - 04/15/2025, 12:32:29 PM     LOG [Microservices:ApiKeyService] Found file at old location

[Nest] 7  - 04/15/2025, 12:32:29 PM   DEBUG [Microservices:ApiKeyService] Attempting to rename file: upload/upload/2d4a1860-8892-45da-9036-64f29b012f2a/7a/4c/7a4c8a0a-8281-4a4e-ab3a-e6bddaefcd80.avif => upload/library/admin/2024/03/test.avif

[Nest] 7  - 04/15/2025, 12:32:29 PM   DEBUG [Microservices:StorageTemplateService] Cleaning up empty directories...

[Nest] 7  - 04/15/2025, 12:32:29 PM     LOG [Microservices:StorageTemplateService] Finished storage template migration

[mmomjian]

Great!

[joojoooo]

I was wondering what would happen if I manually unmounted the drive to simulate an outage and I found a similar issue but with "empty trash" , immich thinks the deletion is successful even if the file is still there. Doesn't seem to try to delete it again when the drive is mounted back

DEBUG [Microservices:TrashService] Queueing 1 asset(s) for deletion from the trash
LOG [Microservices:TrashService] Queued 1 asset(s) for deletion from the trash
DEBUG [Microservices:StorageTemplateService] Cleaning up move history for asset 96a202fe-a4d3-4948-a697-2a60ffd288eb
WARN [Microservices:StorageRepository] File upload/library/e5e65472-71a5-4a45-91ad-c8b098c92010/2025/01/test.avif does not exist.

[bo0tzz]

This is working as expected, if we made it so that a missing file fails the whole delete then it could get stuck in that state and never be able to clear it from the trash.