Can't use any sdxl lora with the OpenCL backend: GGML_ASSERT(src0->type == src1->type) failed

[StellanWay]

I understand that the GGML OpenCL backend has many limitations, however it's much faster than the cpu backend on my phone. Since Conv2D direct sdxl generation now only takes 15-20 seconds per iteration, which is almost usable or would be with an accelerator lora, but I can't seem to be able to load any.

This works with the cpu backend, but not the OpenCL one:

sdcpp --model "sd_xl_base-1.0.safetensors" --vae "sdxl_vae.safetensors" --prompt "<lora:dmd2_sdxl_4step_lora_fp16:1.0>A dumb cat" 

ggml_opencl: selected platform: 'QUALCOMM Snapdragon(TM)'
ggml_opencl: device: 'QUALCOMM Adreno(TM) 830 (OpenCL 3.0 Adreno(TM) 830)'
ggml_opencl: OpenCL driver: OpenCL 3.0 QUALCOMM build: 0800.35 Compiler E031.47.18.28
ggml_opencl: vector subgroup broadcast support: true
ggml_opencl: device FP16 support: true
ggml_opencl: mem base addr align: 128
ggml_opencl: max mem alloc size: 1024 MB
ggml_opencl: SVM coarse grain buffer support: true
ggml_opencl: SVM fine grain buffer support: true
ggml_opencl: SVM fine grain system support: false
ggml_opencl: SVM atomics support: true
ggml_opencl: flattening quantized weights representation as struct of arrays (GGML_OPENCL_SOA_Q)
ggml_opencl: using kernels optimized for Adreno (GGML_OPENCL_USE_ADRENO_KERNELS)
ggml_opencl: loading OpenCL kernels...........................................................
ggml_opencl: default device: 'QUALCOMM Adreno(TM) 830 (OpenCL 3.0 Adreno(TM) 830)'
[INFO ] stable-diffusion.cpp:192  - loading model from 'sd_xl_base_1.0.safetensors'
[INFO ] model.cpp:1013 - load sd_xl_base_1.0.safetensors using safetensors format
[INFO ] stable-diffusion.cpp:231  - loading vae from 'sdxl_vae.safetensors'
[INFO ] model.cpp:1013 - load sdxl_vae.safetensors using safetensors format
[INFO ] stable-diffusion.cpp:243  - Version: SDXL
[INFO ] stable-diffusion.cpp:272  - Weight type:                 f16
[INFO ] stable-diffusion.cpp:273  - Conditioner weight type:     f16
[INFO ] stable-diffusion.cpp:274  - Diffusion model weight type: f16
[INFO ] stable-diffusion.cpp:275  - VAE weight type:             f16
  |==================================================| 2641/2641 - 250.00it/s
  |==================>                               | 959/2641 - 0.00it/s
[INFO ] stable-diffusion.cpp:569  - total params memory size = 6751.89MB (VRAM 6751.89MB, RAM 0.00MB): clip 1757.36MB(VRAM), unet 4900.07MB(VRAM), vae 94.47MB(VRAM), controlnet 0.00MB(VRAM), pmid 0.00MB(VRAM)
[INFO ] stable-diffusion.cpp:573  - loading model from 'sd_xl_base_1.0.safetensors' completed, taking 9.31s
[INFO ] stable-diffusion.cpp:615  - running in eps-prediction mode
[INFO ] stable-diffusion.cpp:2028 - TXT2IMG
[INFO ] stable-diffusion.cpp:749  - Attempting to apply 1 LoRAs
[INFO ] model.cpp:1013 - load dmd2_sdxl_4step_lora_fp16.safetensors using safetensors format
[INFO ] lora.hpp:117  - loading LoRA from 'dmd2_sdxl_4step_lora_fp16.safetensors'
  |==================================================| 2364/2364 - 0.00it/s
/data/data/com.termux/files/home/.termux-build/sdcpp-opencl/src/ggml/src/ggml-opencl/ggml-opencl.cpp:3697: GGML_ASSERT(src0->type == src1->type) failed
0: 0x607554fcc8
1: 0x607554fc78
2: 0x6075561ed8
3: 0x607552ad90
4: 0x607552848c
5: 0x607554ac04
6: 0x6075564aa0
7: 0x6075434d6c
8: 0x60754975d8
9: 0x607541d704
10: 0x60753fddd0
11: 0x6075403768
12: 0x6075386b3c
13: 0x74f114d8e8 __libc_init
The process was killed by SIGABRT: Aborted
The process was killed by SIGABRT: Aborted

@Green-Sky @stduhpf Is this an OpenCL backend issue or am I doing something wrong? I don't have any experience with any of this, that's why I'm asking. If it's an issue is it possible to apply the lora to the model with some other program to create a new model or that's not how it works at all?

[stduhpf]

This looks like an OpenCL backend issue, indeed. Looks like it doesn't support adding different types together. And lora support here involves adding the merged weigthts of the LoRA (I'm pretty sure it's f32 when merged) to the model weights (f16 here). Does --type f32 fix it?

[StellanWay]

This looks like an OpenCL backend issue, indeed. Looks like it doesn't support adding different types together. And lora support here involves adding the merged weigthts of the LoRA (I'm pretty sure it's f32 when merged) to the model weights (f16 here). Does --type f32 fix it?

I get the same error with --type f32 and --type f16.

/data/data/com.termux/files/home/.termux-build/sdcpp-opencl/src/ggml/src/ggml-opencl/ggml-opencl.cpp:3697: GGML_ASSERT(src0->type == src1->type) failed

With --type q4_0 I get a different error.

/data/data/com.termux/files/home/.termux-build/sdcpp-opencl/src/ggml/src/ggml-opencl/ggml-opencl.cpp:6373: GGML_ASSERT(false && "not implemented") failed

On the master branch here vae is forced to be f32 for sdxl for some reason, I've patched that out hoping it would help, but it doesn't work either way.

[rmatif]

This should fix the issue: ggml-org/llama.cpp#15140

Also, use --diffusion-conv-direct. With accelerated LoRA, you can set --cfg-scale 1. Using both options, you should get around 3-4 s/it on Adreno 830.

However, it's more convenient to apply the LoRA before usage. You can do this either with this PR #709 or by using ComfyUI and exporting the model afterward

[StellanWay]

This should fix the issue: ggml-org/llama.cpp#15140

Also, use --diffusion-conv-direct. With accelerated LoRA, you can set --cfg-scale 1. Using both options, you should get around 3-4 s/it on Adreno 830.

However, it's more convenient to apply the LoRA before usage. You can do this either with this PR #709 or by using ComfyUI and exporting the model afterward

Thank you @rmatif, it works. After applying your changes I've managed to generate this abomination using the dmd2 lora.

I still only get 16-20 s/it even with --diffusion-conv-direct and --cfg-scale 1, but that's for 1024x1024.

Unfortunately vae takes a lot of time for 1024x1024 even with --vae-conv-direct, because I have to use --vae-tiling. The OpenCL max allocation size is capped at 1024 MB, so I imagine there is not much that can be done. The Vulkan max allocation size is 2048 MB and I have to set GGML_VK_FORCE_MAX_ALLOCATION_SIZE to make the backend work, but even then it only produces garbage.

What's annoying is that I can't use my phone when generating a picture, because the Android UI freezes. The screen is only updated every few seconds. It's a drawing issue of some kind, not a running out of memory issue (I have a Oneplus 13 with 24 GB RAM).

[wbruna]

You could try #484 , adjusting SD_TILE_SIZE / SD_TILE_OVERLAP to minimize the total number of tiles ( #701 has a few benchmarks, but the results are likely strongly dependent on the hardware / backend).

[rmatif]

I still only get 16-20 s/it even with --diffusion-conv-direct and --cfg-scale 1, but that's for 1024x1024.

My bad, I thought you were inferencing at 512x512. I'm also getting around 15s/it at that resolution. If it's too slow for you, try lowering it a bit, for example, to 768x768, it should be about twice as fast

Unfortunately vae takes a lot of time for 1024x1024 even with --vae-conv-direct, because I have to use --vae-tiling. The OpenCL max allocation size is capped at 1024 MB, so I imagine there is not much that can be done.

Try to use --taesd with taesdxl it will reduce greatly the decode time and you will stay within the 1024 MB limit so no need to vae-tiling

What's annoying is that I can't use my phone when generating a picture, because the Android UI freezes. The screen is only updated every few seconds. It's a drawing issue of some kind, not a running out of memory issue (I have a Oneplus 13 with 24 GB RAM).

That’s a scheduler issue, it doesn’t handle well the workload we’re sending to it.

Additionally, you can take a look at #675, there are some good one-step models supported there