Machine learning request to "http://immich-machine-learning:3003" failed with status 500

[H4rl3k1n]

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

• Yes

The bug

since latest version 1.132.3 searchfunction doesnt work anymore. i tryed different clip models:
immich-app/XLM-Roberta-Large-Vit-B-16Plus
ViT-B-16-SigLIP-i18n-256__webli

The OS that Immich Server is running on

Ubuntu Server 24.04 lts

Version of Immich Server

1..132.3

Version of Immich Mobile App

1.132.3

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, rocm, openvino, rknn] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, rocm, openvino, openvino-wsl, rknn] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - /dockervolumes/immich/immich_cache:/cache
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:148bb5411c184abd288d9aaed139c98123eeb8824c5d3fce03cf721db58066d8
    healthcheck:
      test: redis-cli ping || exit 1
    volumes:
      - /dockervolumes/immich/immich_redis:/data
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:739cdd626151ff1f796dc95a6591b55a714f341c737e27f045019ceabf8e8c52
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: >-
        pg_isready --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" || exit 1; Chksum="$$(psql --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: >-
      postgres -c shared_preload_libraries=vectors.so -c 'search_path="$$user", public, vectors' -c logging_collector=on -c max_wal_size=2GB -c shared_buffers=512MB -c wal_compression=on
    restart: always

Your .env content

UPLOAD_LOCATION=/dockervolumes/immich/immich_upload
DB_DATA_LOCATION=/dockervolumes/immich/immich_postgres
DB_PASSWORD=postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
IMMICH_VERSION=v1.132.3

Reproduction steps

1. upgrade to version 1.132.3
2. use search function

Relevant log output

[Nest] 17  - 05/01/2025, 6:45:55 AM    WARN [Api:MachineLearningRepository~m3pj7itl] Machine learning request to "http://immich-machine-learning:3003" failed with status 500: Internal Server Error

[Nest] 17  - 05/01/2025, 6:45:55 AM   ERROR [Api:ErrorInterceptor~m3pj7itl] Unknown error: Error: Machine learning request '{"clip":{"textual":{"modelName":"ViT-B-16-SigLIP-i18n-256__webli","options":{"language":"de"}}}}' failed for all URLs

Error: Machine learning request '{"clip":{"textual":{"modelName":"ViT-B-16-SigLIP-i18n-256__webli","options":{"language":"de"}}}}' failed for all URLs

    at MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:98:15)

    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)

    at async MachineLearningRepository.encodeText (/usr/src/app/dist/repositories/machine-learning.repository.js:121:26)

    at async SearchService.searchSmart (/usr/src/app/dist/services/search.service.js:62:27)

Additional information

i tryed the usual stuff:
new machine learning intelligent search, reboot, restart, reinstall but always get this error.

i tryed a fallback to 1.131.3, here the searchfunction works fine.

[bo0tzz]

Can you post the logs from the ML container?

[H4rl3k1n]

this container has no problems seems:
...

                         │                                                 │

                         │   225 │   │   │   │   model.load()              │

                         │   226 │   │   │   except FileNotFoundError as e │

                         │   227 │   │   │   │   if model.model_format ==  │

                         │ ❱ 228 │   │   │   │   │   raise e               │

                         │   229 │   │   │   │   log.warning(              │

                         │   230 │   │   │   │   │   f"{model.model_format │

                         │       '{model.model_name}' does not support it. │

                         │   231 │   │   │   │   │   exc_info=e,           │

                         │                                                 │

                         │ /usr/src/immich_ml/main.py:225 in _load         │

                         │                                                 │

                         │   222 │   │   │   raise HTTPException(500, f"Fa │

                         │   223 │   │   with lock:                        │

                         │   224 │   │   │   try:                          │

                         │ ❱ 225 │   │   │   │   model.load()              │

                         │   226 │   │   │   except FileNotFoundError as e │

                         │   227 │   │   │   │   if model.model_format ==  │

                         │   228 │   │   │   │   │   raise e               │

                         │                                                 │

                         │ /usr/src/immich_ml/models/base.py:51 in load    │

                         │                                                 │

                         │    48 │   │   │   return                        │

                         │    49 │   │   self.load_attempts += 1           │

                         │    50 │   │                                     │

                         │ ❱  51 │   │   self.download()                   │

                         │    52 │   │   attempt = f"Attempt #{self.load_a │

                         │       else "Loading"                            │

                         │    53 │   │   log.info(f"{attempt} {self.model_ │

                         │       '{self.model_name}' to memory")           │

                         │    54 │   │   self.session = self._load()       │

                         │                                                 │

                         │ /usr/src/immich_ml/models/base.py:44 in         │

                         │ download                                        │

                         │                                                 │

                         │    41 │   │   │   log.info(                     │

                         │    42 │   │   │   │   f"Downloading {self.model │

                         │       '{self.model_name}'. This may take a whil │

                         │    43 │   │   │   )                             │

                         │ ❱  44 │   │   │   self._download()              │

                         │    45 │                                         │

                         │    46 │   def load(self) -> None:               │

                         │    47 │   │   if self.loaded:                   │

                         │                                                 │

                         │ /usr/src/immich_ml/models/base.py:76 in         │

                         │ _download                                       │

                         │                                                 │

                         │    73 │   │   │   ModelFormat.RKNN: ["*.armnn"] │

                         │    74 │   │   }                                 │

                         │    75 │   │                                     │

                         │ ❱  76 │   │   snapshot_download(                │

                         │    77 │   │   │   f"immich-app/{clean_name(self │

                         │    78 │   │   │   cache_dir=self.cache_dir,     │

                         │    79 │   │   │   local_dir=self.cache_dir,     │

                         │                                                 │

                         │ /opt/venv/lib/python3.11/site-packages/huggingf │

                         │ ace_hub/utils/_validators.py:114 in _inner_fn   │

                         │                                                 │

                         │   111 │   │   if check_use_auth_token:          │

                         │   112 │   │   │   kwargs = smoothly_deprecate_u │

                         │       has_token=has_token, kwargs=kwargs)       │

                         │   113 │   │                                     │

                         │ ❱ 114 │   │   return fn(*args, **kwargs)        │

                         │   115 │                                         │

                         │   116 │   return _inner_fn  # type: ignore      │

                         │   117                                           │

                         │                                                 │

                         │ /opt/venv/lib/python3.11/site-packages/huggingf │

                         │ ace_hub/_snapshot_download.py:235 in            │

                         │ snapshot_download                               │

                         │                                                 │

                         │   232 │   │   │   raise api_call_error          │

                         │   233 │   │   else:                             │

                         │   234 │   │   │   # Otherwise: most likely a co │

                         │       the user                                  │

                         │ ❱ 235 │   │   │   raise LocalEntryNotFoundError │

                         │   236 │   │   │   │   "An error happened while  │

                         │       cannot find the appropriate"              │

                         │   237 │   │   │   │   " snapshot folder for the │

                         │       check your internet connection"           │

                         │   238 │   │   │   │   " and try again."         │

                         ╰─────────────────────────────────────────────────╯

                         LocalEntryNotFoundError: An error happened while   

                         trying to locate the files on the Hub and we cannot

                         find the appropriate snapshot folder for the       

                         specified revision on the local disk. Please check 

                         your internet connection and try again.            

[05/01/25 06:50:55] INFO Shutting down due to inactivity.

[05/01/25 06:50:55] INFO Shutting down

[05/01/25 06:50:55] INFO Waiting for application shutdown.

[05/01/25 06:50:55] INFO Application shutdown complete.

[05/01/25 06:50:55] INFO Finished server process [9]

[05/01/25 06:50:55] ERROR Worker (pid:9) was sent SIGINT!

[05/01/25 06:50:55] INFO Booting worker with pid: 98

[05/01/25 06:50:57] INFO Started server process [98]

[05/01/25 06:50:57] INFO Waiting for application startup.

[05/01/25 06:50:57] INFO Created in-memory cache with unloading after 300s

                         of inactivity.                                     

[05/01/25 06:50:57] INFO Initialized request thread pool with 4 threads.

[05/01/25 06:50:57] INFO Application startup complete.

[bo0tzz]

It looks to me like there was definitely an error happening, but the logs are incomplete.

[aliowx]

What's going on!?