FACIAL RECOGNITION job runs everyday to process same set of photos

[ufizo]

The bug

My server is currently hosting about 149776 photos and 10447 videos. Every morning at 8AM in my timezone (UTC +8:00), I hear the server fans get loud and upon checking I can see FACIAL RECOGNITION job is running with about 50000 assets in waiting. If I allow it to run, it takes about 6 hours to complete.
The problem is that this runs for 6 hours everyday. I expected once the existed assets are processed, they should not be in the queue to be processed everyday.

Further, my cron expression for periodic scanning is: 0 18 * * 6
I have set this because I want it to only run once in a week, however something is still triggering facial recognition everyday at 00:00 UTC.

There are two issues:

1. Why a third of my library is processed everyday for facial recognition?
2. If the cron expression is getting ignored? This is something that has started to happen after upgrading to v1.100.0

I am happy to provide any additional information or logs that may help to identify the cause.

The OS that Immich Server is running on

Arch Linux

Version of Immich Server

v1.100.0

Version of Immich Mobile App

v1.100.0

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

version: "3.8"

name: immich

services:
  immich-server:
    container_name: immich_server
    user: ${PUID}:${PGID}
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      #- ${THUMBS_LOCATION}:/usr/src/immichthumbs
      - ${EXT_LIB}:/mnt/media:ro
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always
    networks:
      - traefik-net
    labels:
      # Explicitly instruct Traefik to expose this service
      - traefik.enable=true
      - traefik.http.routers.immich-api.entrypoints=websecure
      - traefik.http.routers.immich-api.tls=true
      - traefik.http.routers.immich-api.tls.certresolver=production
      # Service configuration
      - traefik.http.services.immich-api.loadbalancer.server.port=3001

  immich-microservices:
    container_name: immich_microservices
    user: ${PUID}:${PGID}
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      #- ${THUMBS_LOCATION}:/usr/src/immichthumbs
      - ${EXT_LIB}:/mnt/media:ro
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always
    networks:
      - traefik-net

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    networks:
      - traefik-net

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:b6124ab2e45cc332e16398022a411d7e37181f21ff7874835e0180f56a09e82a
    restart: always
    networks:
      - traefik-net

  database:
    container_name: immich_postgres
    image: tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    env_file:
      - .env
    ports:
      - 5432:5432
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always
    networks:
      - traefik-net

volumes:
  pgdata:
  model-cache:

networks:
  traefik-net:
    external: true

Your .env content

PUBLIC_LOGIN_PAGE_MESSAGE="****"
#PUBLIC_IMMICH_SERVER_URL="****"
REVERSE_GEOCODING_DUMP_DIRECTORY=/tmp/immich-reverse-geocoding-dump

# The location where your uploaded files are stored
UPLOAD_LOCATION=/mnt/immich/UPLOAD_LOCATION/
EXT_LIB=/mnt/immich/ext-lib

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=****

# RUN as user
PUID=1000
PGID=1000

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

Reproduction steps

1. Check server status at 8 AM everyday
2. CPU is at high utilization with high temps
3. Immich is running FACIAL RECOGNITION job that continues for 6 hours

Additional information

Immich is the only application running on the server.
Utilization is low, there are only a few users and we take <10 pictures a week.

[mertalev]

The cron setting is specifically for library maintenance; there are other tasks that run daily regardless. Regarding the facial recognition speed, 6 hours is unusually long so I'd like to ask a few questions if you don't mind:

1. What are the hardware specs of the server? Most relevant are CPU, RAM and the disk and file system that Postgres is running on.

2. Are all of the facial recognition settings default, or did you change any? There isn't anything wrong with changing them - this is just to get a better idea.

3. With a SQL client, can you query and share the output of SELECT * FROM pg_vector_index_stat;?

[ufizo]

Here are my responses:

1. It is running on a VM - CPU: 4 cores, 8 threads (AMD Ryzen 5 5600), RAM: 4GB, Filesystem: SMB mount of a NAS in LAN.

2. Here are my settings:

   "facialRecognition": {
      "enabled": true,
      "modelName": "buffalo_l",
      "minScore": 0.7,
      "maxDistance": 0.5,
      "minFaces": 10
    }

3. No data

 tablerelid | indexrelid | tablename | indexname | idx_status | idx_indexing | idx_tuples | idx_sealed | idx_growing | idx_write | idx_size | idx_options 
------------+------------+-----------+-----------+------------+--------------+------------+------------+-------------+-----------+----------+-------------
(0 rows)

Thank you for taking the time to review my case.

[mertalev]

That's weird, you don't have any vector indices. You can run these to make things much faster:

SET vectors.pgvector_compatibility=on;

CREATE INDEX IF NOT EXISTS clip_index ON smart_search
    USING hnsw (embedding vector_cosine_ops)
    WITH (ef_construction = 300, m = 16);

CREATE INDEX IF NOT EXISTS face_index ON asset_faces
    USING hnsw (embedding vector_cosine_ops)
    WITH (ef_construction = 300, m = 16);

[ufizo]

Both create commands return:
ERROR: internal error: entered unreachable code

[barrancoguy]

I'm here because every 2 days the facial recognition process rescan all my photos. That's is weird , right ? . I try ro reset the process but the weird rescan continues without a reason.

[mertalev]

It runs every night unless there were no new assets since the last time it ran. It's part of the clustering algorithm. It shouldn't take too long or use too much resources, though. Do you have the facial recognition settings at default?

[sorinvasilescu]

I have the same problem. It starts running facial recognition on some tens of thousands of photos each night at 12 am, even though no new photos were added, and it takes hours.

This seems to be what it's running:
with "cte" as (select "asset_faces"."id", "asset_faces"."personId", face_search.embedding <=> $1 as "distance" from "asset_faces" inner join "assets" on "assets"."id" = "asset_faces"."assetId" inner join "face_search" on "face_search"."faceId" = "asset_faces"."id" left join "person" on "person"."id" = "asset_faces"."personId" where "assets"."ownerId" = any($2::uuid[]) and "assets"."deletedAt" is null and ("person"."birthDate" is null or "person"."birthDate" <= $3) order by face_search.embedding <=> $4 limit $5) select * from "cte" where "cte"."distance" <= $6

[mertalev]

Did you restore from a backup? The facial recognition index makes this quick, but it sometimes doesn't get created during restores. The answer still applies today, except that asset_faces should be replaced with face_search.

[sorinvasilescu]

The answer still applies today, except that asset_faces should be replaced with face_search.

Yes, figured it out from the code. :)

Yes, I was missing the face_index. My bad. The query went from 1.3 seconds to 67ms.
Yes, I did restore from backup.

Thanks for the quick reply, though!