Immich database reset after update

[alexperjescu]

Hi,

I have Immich running on Portainer. Most of the time updates are smooth. But sometimes, even when there are no breaking changes, Immich completely resets after updating, I get the "Welcome to Immich. Getting Started" message.

The last time I updated from 1.126.1 to 1.128.0 I encountered the same problem. Usually I can restore the database from backup and continue, however I have a 2.8TB library so it is not a quick process. Also, in the past, I sometimes encountered issues when restoring the database from backup, sometimes Immich would loop through the machine learning processes.

Any ideas on why this happens? Is Portainer an issue? Thanks for your input.

`name: immich

services:
immich-server:
container_name: immich_server
image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
# extends:
# file: hwaccel.transcoding.yml
# service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
volumes:
- immichData:/usr/src/app/upload
- /etc/localtime:/etc/localtime:ro
env_file:
- stack.env
ports:
- 2283:2283
depends_on:
- redis
- database
restart: always

immich-machine-learning:
container_name: immich_machine_learning
# For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
# Example tag: ${IMMICH_VERSION:-release}-cuda
image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
# extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
# file: hwaccel.ml.yml
# service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the -wsl version for WSL2 where applicable
volumes:
# - model-cache:/cache
- immichRedis:/cache
env_file:
- stack.env
restart: always

redis:
container_name: immich_redis
image: docker.io/redis:6.2-alpine@sha256:328fe6a5822256d065debb36617a8169dbfbd77b797c525288e465f56c1d392b
healthcheck:
test: redis-cli ping || exit 1
restart: always

database:
container_name: immich_postgres
image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
environment:
POSTGRES_PASSWORD: ${DB_PASSWORD}
POSTGRES_USER: ${DB_USERNAME}
POSTGRES_DB: ${DB_DATABASE_NAME}
POSTGRES_INITDB_ARGS: '--data-checksums'
volumes:
- ${DB_DATA_LOCATION}:/var/lib/postgresql/data
healthcheck:
test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
interval: 5m
start_interval: 30s
start_period: 5m
command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
restart: always

volumes:
immichData:
external: true
immichRedis:
external: true
immichDatabase:
external: true
model-cache:`

[bo0tzz]

What is your DB_DATA_LOCATION set to, and what sort of storage is that on etc?

[alexperjescu]

The database is stored on the local volume

DB_DATA_LOCATION=./postgres

[akostadinov]

Are you sure this is persisted between restart of the database container? Just asking. I had an issue where I had a typo in the mount path so database was wiped up after restart.

[alexperjescu]

Thanks for the advice @akostadinov! Yes, the volume is persistent, 3 out of 4 upgrades are successful, only every so often the database is lost (it's not a matter of updating the docker compose file either, I'm talking about minor updates).

Also, restarting that Portainer instance does not affect the database.

[akostadinov]

When I upgrade immich, I only restart server and machine learning containers. Idk how is portainer working. But never so far I had database lost by updating immich.

What happens if you just restart immich? Can it be reproduced just like this?

I'm not an expert in immich but it very much sounds like your runtime is clearing your database storage. I very much doubt immich has any kind code to remove existing database. I haven't looked into the code. But again, unlikely to have such code.

Maybe you can enable query log in your database as see what's going on. Whether commands to delete anything is being run. Or just the data is lost before immich has started.

[scorninpc]

i'm facing this problem today

updating to IMMICH_VERSION=release cause database cleanup on first run

I restore database, start docker and it's clean. create user, restart and not clean anymore. I think this clean on migration scripts

I dont know why, but force IMMICH_VERSION=v1.131.3 stop this problem (i dont know why, because 1.131.3 is the release version)

well, to fix, i just force IMMICH_VERSION=v1.131.3 at .env. Restore database backup, and start container. All works fine now

thanks for automated daily backups =D