add_raster doesn't work in container environment

[ldemaz]

Environment Information

• leafmap version: 0.42.6
• Python version: 3.11.6
• Operating System: Mac OS 14.6.1 (but running container giwqs/leafmap:latest)

Description

I am having difficult getting leafmap to display a local COG when running in a container. I have tried multiple permutations of building leafmap into a docker container, all of which show slightly different failures. These are described below.

What I Did

I will start with the most recent, which is simply directly running the image giwqs/leafmap:latest, with the following command:

docker run -it -p 8888:8888 -v $(pwd):/home/jovyan/work -v /Users/me/images/tiles:/home/tiles giswqs/leafmap:latest

Note I am mounting a few directories here to get access to local files, the most important of which is a directory containing the local COGs I want to display, which I form paths to and then try to display using these lines, adapted from several notebook cells:

import os
import sys
import pandas as pd
import re
import numpy as np
import leafmap.leafmap as leafmap

# traverse directory containing COGs to make catalog
paths = []
for root, _, files in os.walk("/home/tiles/", topdown=True):
    for f in files:
        if f.endswith(".tif"):
            file_bits = f.split("_")
            file_dict = {
                "tile": int(re.sub("tile", "", file_bits[0])),
                "year": file_bits[1].split("-")[0],
                "month": file_bits[1].split("-")[1],
                "file": f,
                "path": os.path.join(root, f)
            }
            paths.append(file_dict)

image_catalog = pd.DataFrame(paths)

# pluck out tile of interest 
image_catalogr = (
    image_catalog[image_catalog['tile']==842099]
    .groupby(["tile", "year"])
    .first()
)
tile = image_catalogr.path.iloc[0]

# display
m = leafmap.Map(zoom=20, 
    center=[pt.y.iloc[0], pt.x.iloc[0]]
)
m.add_basemap("SATELLITE")
m.add_raster(tile, bands=[1,2,3], layer_name='TRUE COLOR')
m

The first failure is that add_raster says xarray isn't installed. To fix this, I simply opened a terminal in the jupyter lab environment and ran mamba install xarray, restarted the kernel, and ran again.

This time it ran, but the COG does not display. Going on previous guidance, I then also tried adding jupyter-server-proxy and then added the following to lines to my imports:

import localtileserver
os.environ['LOCALTILESERVER_CLIENT_PREFIX'] = '/proxy/{port}'

Restarted and ran again. The COG still doesn't show up.

So I am stuck here. I should note I have also tried to build my own containers, using several different variants. One of these is as follows, which is simply adapting the Dockerfile in this repo, and adding the missing xarray and rioxarray, and a few other bits:

FROM jupyter/scipy-notebook:latest
RUN mamba install -c conda-forge leafmap geopandas "localtileserver>=0.10.0" osmnx -y && \
    pip install -U leafmap jsonschema==4.18.0 lonboard h5py xarray==2024.11.0 rioxarray==0.17.0 && \
    fix-permissions "${CONDA_DIR}" && \
    fix-permissions "/home/${NB_USER}"


ENV PROJ_LIB='/opt/conda/share/proj'

USER root

# Set up working directory
RUN mkdir -p /home/workdir
WORKDIR /home/workdir

# Activate the Conda environment and ensure it's available in the container
ENV PATH="/opt/conda/base:$PATH"
ENV CONDA_DEFAULT_ENV=geo

# Expose Jupyter Lab's default port
EXPOSE 8888

# Run Jupyter Lab 
ENTRYPOINT ["jupyter", "lab", "--ip=0.0.0.0", "--port=8888", "--no-browser", "--allow-root"]

Running the same code as above, I get a different error, which has bedevilled me across multiple attempts at this, which is that I get an error saying localtileserver doesn't exist, when it does, but it is failing because:

ModuleNotFoundError: No module named 'rio_tiler.io'

A different, pip-based version of the build script bypasses the error:

# build script fixed and optimized by ChatGPT
FROM continuumio/miniconda3:24.9.2-0

# Use a single RUN command where possible for efficiency
RUN apt-get update && \
    apt-get --allow-releaseinfo-change update && \
    apt-get --allow-releaseinfo-change-suite update && \
    apt-get install -y binutils libproj-dev gdal-bin libgdal-dev g++ && \
    rm -rf /var/lib/apt/lists/*

# solution from here: https://stackoverflow.com/a/73101774
RUN cp /usr/lib/aarch64-linux-gnu/libstdc++.so.6.0.30 /opt/conda/lib/ && \
    cd /opt/conda/lib/ && \
    rm -f libstdc++.so.6 && \
    ln -s libstdc++.so.6.0.30 libstdc++.so.6

# Upgrade pip, pip-tools, and setuptools
RUN pip install --no-cache-dir --upgrade pip pip-tools setuptools

# Copy requirements.txt and install dependencies
COPY requirements.txt /tmp/requirements.txt
RUN pip install --no-cache-dir -r /tmp/requirements.txt && \
    rm /tmp/requirements.txt

# Set up working directory
RUN mkdir -p /home/workdir
WORKDIR /home/workdir

# Expose Jupyter Lab's default port
EXPOSE 8888

# Define entrypoint with corrected Jupyter Lab options
ENTRYPOINT ["jupyter", "lab", "--ip=0.0.0.0", "--port=8888", "--no-browser", "--allow-root"]

With requirements:

jupyterlab
ipywidgets==8.1.5
ipyleaflet==0.19.2
leafmap==0.41.2
localtileserver==0.10.5
numpy==2.1.3
geopandas==1.0.1
pandas==2.2.3
shapely==2.0.6
matplotlib==3.9.2

But this has the same problem with displaying COGs--they just don't show up (and I have tried adding jupyter-server-proxy).

So I am stuck here. I will not that an ordinary, non containerized install of leafmap using conda/mamba does show the COGs, but I want to containerize this for use on a cluster account.

This is probably an issue with localtileserver, but perhaps there is a fix known here for getting it to work with add_raster. Any solutions will be most appreciated.

[giswqs]

Take a look at this one to see if it helps.
banesullivan/localtileserver#179 (comment)

[ldemaz]

Thanks for the suggestion. Unfortunately not. I tried the suggested Dockerfile, including with various permutations, but it hangs up on gdal. Build logs show an error related to not finding gdal-config:

Could not find gdal-config. Make sure you have installed the GDAL native library and development headers.
[end of output]

[ldemaz]

Note that adding the following:

RUN apt-get update &&\
    apt-get install -y binutils libproj-dev gdal-bin

To the Dockerfile first doesn't solve it (after changing from USER jovyan to root first)

[ldemaz]

Just as an update to this, add_raster (and thus presumably localtileserver) is also not working on a jupyter lab session running on an HPC cluster, even from within the same conda environment that I used successfully locally. This was set up as:

mamba create -n geo leafmap geopandas localtileserver python xarray rioxarray -c conda-forge

My notebook runs fine (accessed on the cluster through an ssh tunnel) and creates a leafmap, but the COGS I am trying to display with add_raster do not show up (I can see they flash up quickly but disappear).

I see from previous issues (here and more recently here) that for running on remote servers one should add:

os.environ['LOCALTILESERVER_CLIENT_PREFIX'] = 'proxy/{port}'

After installing jupyter-server-proxy, but that isn't solving it.

[niowniow]

(I can see they flash up quickly but disappear).

This could actually be a symptom of the issue (#1041) I just submitted, i.e. the localtileserver is shutdown immediately after being started, because the tileclient is is deleted (or garbage collected) in leafmap.

[DWGodwin]

I was having this issue with add_raster running leafmap notebooks in a docker container on a remote server and managed to fix it for my case with a modification to the leafmap Dockerfile. My issue was that using add_raster did result in the raster being listed in the layers in the leafmap widget, but the raster was not being visualized in the widget.

I was trying to address this by installing jupyter-server-proxy via pip in the jupyter lab terminal and running os.environ['LOCALTILESERVER_CLIENT_PREFIX'] = 'proxy/{port}' in the notebook based on advice here, but did not see any changes in the result.

The key for my case was installing jupyter-server-proxy and xarray_leaflet in the Dockerfile and adding the env variable ENV LOCALTILESERVER_CLIENT_PREFIX='proxy/{port}'. Here is the Dockerfile I used to fix this issue, modified from the repository Dockerfile - I hope this helps anyone experiencing the same problem.

FROM jupyter/scipy-notebook:latest
RUN mamba install -c conda-forge leafmap geopandas "localtileserver>=0.10.0" osmnx -y && \
    pip install -U leafmap jsonschema==4.18.0 jupyter-server-proxy==4.4.0 xarray_leaflet==0.2.3 lonboard h5py && \
    fix-permissions "${CONDA_DIR}" && \
    fix-permissions "/home/${NB_USER}"

ENV PROJ_LIB='/opt/conda/share/proj'
ENV LOCALTILESERVER_CLIENT_PREFIX='proxy/{port}'

USER root
RUN chown -R ${NB_UID} ${HOME}
USER ${NB_USER}

[giswqs]

@DWGodwin Thank you for sharing your solution. xarray_leaflet has been deprecated a while ago and is not used by leafmap any more. Have you tried removing xarray_leaflet and see if add_raster still works?

[DWGodwin]

@giswqs I can confirm that using substituting xarray for xarray_leaflet works, here is a slightly tweaked Dockerfile I used to display rasters on a remote server:

FROM jupyter/scipy-notebook:latest
RUN mamba install -c conda-forge leafmap geopandas "localtileserver>=0.10.0" osmnx -y && \
    pip install -U leafmap jsonschema==4.18.0 jupyter-server-proxy==4.4.0 xarray lonboard h5py && \
    fix-permissions "${CONDA_DIR}" && \
    fix-permissions "/home/${NB_USER}"

ENV PROJ_LIB='/opt/conda/share/proj'
ENV LOCALTILESERVER_CLIENT_PREFIX='proxy/{port}'

USER root
RUN chown -R ${NB_UID} ${HOME}
USER ${NB_USER}

[giswqs]

@DWGodwin Awesome! Thanks for sharing.

@ldemaz When you have time, please give it a try and see if it works for you. Sorry for the long delay. I am not an active docker user and have not been able to figure this out on my own.

[giswqs]

I have updated the leafmap Dockerfile in #1044