Meson build, replacing setuptools

Limitation: The sdist (source distribution for PyPI) now contains the
whole of mmCoreAndDevices and more (33 MiB). While technically
functional, we should fix this before merging this change.

Update mmCoreAndDevices to latest (which has meson.build for MMCore and
MMDevice).

Add Meson build file.

Use meson-python so that `python -m build` just works via
pyproject.toml.

Move the single source of truth for the version number from _version.py
(now generated) to meson.build.

The Meson build has several advantages:
- Build details of MMDevice and MMCore come from their own build files,
  rather than being duplicated here in setup.py
- Editable installs truly work (even if C++ files are edited) (Caveat:
  beware of importing pymmcore from the source root)
- Since we are using a true C++ build system, controlling build options
  is much easier and cleaner than it was with setuptools

The main disadvantage is that MANIFEST.in can no longer be used to
control what gets included in the sdist (meson-python uses `meson dist`
to produce the sdist, which includes all version-controlled files).
However, once we are ready to use MMDevice and MMCore from independent
repositories, this will no longer be an issue (and ends up being simpler
than the error-prone MANIFEST.in).

Author: marktsuchida

.github/workflows/ci.yml

@@ -14,18 +14,6 @@ concurrency:
   cancel-in-progress: true
 
 jobs:
-  # check that sdist contains all files and that extra files
-  # are explicitly ignored in manifest or pyproject
-  check-manifest:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          fetch-depth: 0
-          submodules: "recursive"
-      - name: Check manifest
-        run: pipx run check-manifest
-
   test:
     name: Test ${{ matrix.os }} py${{ matrix.python-version }} np${{ matrix.numpy }}
     runs-on: ${{ matrix.os }}
@@ -53,6 +41,8 @@ jobs:
         with:
           submodules: "recursive"
 
+      - uses: ilammy/msvc-dev-cmd@v1
+
       - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v2
         with:
@@ -86,14 +76,12 @@ jobs:
         with:
           submodules: "recursive"
 
+      - uses: ilammy/msvc-dev-cmd@v1
+
       - name: Build wheels
         uses: pypa/[email protected]
         env:
           CIBW_ARCHS_MACOS: "${{ matrix.macos_arch }}"
-          # Python on Linux is usually configured to add debug information,
-          # which increases binary size by ~11-fold. Remove for the builds we
-          # distribute.
-          CIBW_ENVIRONMENT_LINUX: "LDFLAGS=-Wl,--strip-debug"
 
       - uses: actions/upload-artifact@v3
         with:
@@ -110,8 +98,7 @@ jobs:
 
       - name: Build sdist
         run: |
-          pip install -U pip build check-manifest
-          check-manifest
+          pip install -U pip build
           python -m build --sdist
 
       - uses: actions/upload-artifact@v3

.gitignore

@@ -5,15 +5,7 @@ venv/
 
 build/
 dist/
-*.pdb
-*.py[cod]
-
 wheelhouse/
 
-src/pymmcore/pymmcore_swig_wrap.h
-src/pymmcore/pymmcore_swig_wrap.cpp
-src/pymmcore/_pymmcore_swig.*
-src/pymmcore/pymmcore_swig.py
-pymmcore.egg-info
 .mypy_cache/
 

MANIFEST.in

@@ -28,9 +28,10 @@ maintaining separate branches; this can ease transition when the device
 interface version changes. Such branches should be named `mmcore-x.y.z.w`.
 
 When upgrading the MMCore version (by bumping the mmCoreAndDevices submodule
-commit), the pymmcore version in `_version.py` should be updated in synchrony.
-The versioning for the python package is taken dynamically from that file
-in the `[tool.setuptools.dynamic]` table in `pyproject.toml`.
+commit), the pymmcore version in `meson.build` should be updated in synchrony.
+The versioning for the python package is taken dynamically from that file, via
+the generated `_version.py` and the `project.dynamic` field in
+`pyproject.toml`.
 
 ## Building Binary Wheels and Source Distributions
 
@@ -63,12 +64,14 @@ The package can be built in a few ways:
     This will build an sdist and wheel for the current platform and Python
     version, and place them in the `dist` directory.
 
-3. Use `pip install -e .`
+3. Use `pip install --no-build-isolation -e .`
    This will build the extension module in-place and allow you to run tests,
-   but will not build a wheel or sdist.  Note that if you do this, you will
-   need to rerun it each time you change the extension module.
-
-
+   but will not build a wheel or sdist. `meson-python` (the build backend) will
+   arrange to automatically rebuild the extension module each time it is
+   imported. This method requires that you first manually install all of the
+   build requirements listed in `pyproject.toml`. See the
+   [meson-python docs](https://meson-python.readthedocs.io/en/latest/how-to-guides/editable-installs.html)
+   for more information.
 
 ## Release procedure
 
@@ -79,11 +82,11 @@ prefixed to the version:
 ```bash
 git checkout main
 
-vim src/pymmcore/_version.py  # Remove .dev0
+vim meson.build  # Remove .dev0
 git commit -a -m 'Version 1.2.3.42.4'
 git tag -a v1.2.3.42.4 -m Release
 
-vim src/pymmcore/_version.py  # Set version to 1.2.3.42.5.dev0
+vim meson.build  # Set version to 1.2.3.42.5.dev0
 git commit -a -m 'Version back to dev'
 
 git push upstream --follow-tags
@@ -101,8 +104,9 @@ and the binary wheels attached.
 
 - The minimum version of python supported is declared in `pypyproject.toml`,
   in the `[project.requires-python]` section.
-- SWIG 4.x is required and automatically fetched via `pyproject.toml` under
-  `[build-system.requires]`.
+- Meson (via `meson-python`), Ninja, and SWIG 4.x are required and
+  automatically fetched via `pyproject.toml` under `[build-system.requires]`.
+- A C++ toolchain is required and must be available on your system.
 - The build-time versions of numpy are in `pyproject.toml`, in the
   `[build-system.requires]` section.
 - The run-time numpy dependency is declared in `pyproject.toml`, in the
@@ -111,7 +115,7 @@ and the binary wheels attached.
   determined by the settings in the `[tool.cibuildwheel]` section of
   `pyproject.toml`.
 - _We_ should provide wheels for all Python versions we claim to support,
-  built agains the oldest NumPy version that we claim to support. Thus, any
+  built against the oldest NumPy version that we claim to support. Thus, any
   issue with the build or our CI will limit the lowest supported versions.
 
 ## ABI Compatibility
@@ -125,18 +129,6 @@ and the binary wheels attached.
   [`oldest-supported-numpy`](https://github.com/scipy/oldest-supported-numpy)
   in our build requires.
 
-## Building with debug symbols on Windows
-
-Since there is no easy way to pass compile and linker options to `build_clib`,
-the easiest hack is to edit the local `setuptools` installation's
-`_distutils/_msvccompiler.py` to add the compiler flag `/Zi` and linker flag
-`/DEBUG:FULL` (see the method `initialize`). This produces `vc140.pdb`.
-
-(The "normal" method would be to run `setup.py build_clib` and `setup.py
-build_ext` with the `--debug` option, and run with `python_d.exe`. But then we
-would need a debug build of NumPy, which is hard to build on Windows.)
-
-
 ### Legacy Build Notes
 
 Many of these notes are probably obviated by the use of cibuildwheel... but

maintainer-notes.md

@@ -0,0 +1,82 @@
+# Copyright 2020-2024 Board of Regents of the University of Wisconsin System
+#
+# This library is free software; you can redistribute it and/or modify it under
+# the terms of the GNU Lesser General Public License, version 2.1, as published
+# by the Free Software Foundation.
+#
+# This library is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more
+# details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with this library; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+#
+# Author: Mark A. Tsuchida
+
+project(
+    'pymmcore',
+    'cpp',
+    version: '11.1.1.71.1.dev0',
+    meson_version: '>=1.3.0',
+    default_options: [
+        'cpp_std=c++14',
+        'warning_level=3',
+    ],
+    # Until MMDevice and MMCore are available individually, we need to use them
+    # from the same git submodule, so this is a bit of a hack:
+    subproject_dir: 'mmCoreAndDevices',
+)
+
+cxx = meson.get_compiler('cpp')
+if cxx.get_id() in ['gcc', 'clang']
+    add_project_arguments('-Wno-deprecated', language: 'cpp')        # throw()
+    # Disable warnings triggered by SWIG-generated code:
+    add_project_arguments('-Wno-unused-parameter', language: 'cpp')
+    add_project_arguments('-Wno-unused-variable', language: 'cpp')
+endif
+if cxx.get_id() in ['msvc', 'clang-cl']
+    add_project_arguments('-DNOMINMAX', language: 'cpp')
+    add_project_arguments('-D_CRT_SECURE_NO_WARNINGS', language: 'cpp')
+    # Disable warnings triggered by SWIG-generated code:
+    add_project_arguments('/wd4100', language: 'cpp')
+    add_project_arguments('/wd4101', language: 'cpp')
+    add_project_arguments('/wd4127', language: 'cpp')
+    add_project_arguments('/wd4456', language: 'cpp')
+    add_project_arguments('/wd4706', language: 'cpp')
+endif
+
+fs = import('fs')
+
+python = import('python').find_installation(pure: false)
+
+threads_dep = dependency('threads')
+
+numpy_abs_incdir = run_command(
+    python, '-c', 'import numpy; print(numpy.get_include())',
+    check: true,
+).stdout().strip()
+# The "correct" way would be to "detect" NumPy as a dependency. Since we are
+# cutting corners, we need to use a relative path as if the NumPy headers are
+# part of this project.
+numpy_incdirs = include_directories(fs.relative_to(numpy_abs_incdir, '.'))
+
+swig = find_program('swig', native: true)
+
+# For now, use MMCore as a subproject. This may be changed to using as a
+# proper dependency via a wrap, but that will likely require better SWIG
+# support by Meson in order to get the SWIG include directories from the
+# dependency object.
+mmcore_proj = subproject(
+    'MMCore',
+    default_options: {
+        'default_library': 'static',
+        'tests': 'disabled',  # Avoid Catch2 subproject in sdist
+    },
+)
+mmcore_dep = mmcore_proj.get_variable('mmcore')
+
+swig_include_dirs = mmcore_proj.get_variable('swig_include_dirs')
+
+subdir('src/pymmcore')

meson.build

@@ -1,7 +1,8 @@
 # https://peps.python.org/pep-0517/
 [build-system]
 requires = [
-    "setuptools >=61.0.0",
+    "meson-python",
+    "ninja",
     "swig >=4.1",
     # https://github.com/scipy/oldest-supported-numpy/blob/main/setup.cfg
     "numpy==1.19.3; python_version=='3.8' and platform_machine=='aarch64' and platform_python_implementation != 'PyPy'",
@@ -13,7 +14,7 @@ requires = [
     # https://numpy.org/devdocs/dev/depending_on_numpy.html#adding-a-dependency-on-numpy
     "numpy>=2.0.0b1; python_version>='3.9'",
 ]
-build-backend = "setuptools.build_meta"
+build-backend = "mesonpy"
 
 # https://peps.python.org/pep-0621/
 [project]
@@ -46,15 +47,10 @@ test = ["pytest"]
 homepage = "https://micro-manager.org"
 repository = "https://github.com/micro-manager/pymmcore"
 
-[tool.setuptools.dynamic]
-version = { attr = "pymmcore._version.__version__" }
-
-[tool.setuptools.package-dir]
-"" = "src"
-
-[tool.setuptools.package-data]
-"*" = ["py.typed", ".pyi"]
-
+[tool.meson-python.args]
+setup = ['-Dstrip=true']
+install = ['--tags=python-runtime,runtime']
+dist = ['--include-subprojects']
 
 [tool.cibuildwheel]
 # Skip 32-bit builds, musllinux, and PyPy wheels on all platforms