Python versions in the Spark Connect client and server are different

[FristrisX]

Hi!

After the v0.3.0 update I am getting the follow error messages:

This is the top-level error message (I don't think this is this actual issue):
[PARSE_SYNTAX_ERROR] Syntax error at or near 'rating': extra input 'rating'. SQLSTATE: 42601

But digger further in I can find this error message:
Python versions in the Spark Connect client and server are different

Reverting back to v0.2.0 fixes the issue.

Does anyone know how to fix this issue to what could cause it? This occurs when trying to run this as a Notebook and a .py file (packaged as a .whl file)

Thanks,
Tristen

[FristrisX]

Also - I ran the following script in a Notebook (it used serverless compute):

import sys
spark.sql('select current_version()').show()
print (spark.version)
print(sys.version)

Results

+--------------------+
|   current_version()|
+--------------------+
|{16.2.x-photon-sc...|
+--------------------+

3.5.2
3.11.10 (main, Sep  7 2024, 18:35:41) [GCC 11.4.0]

Then I compared it to DQX 0.3.0 configurations:

cli = ["pyspark~=3.5.0"] -- different from the 3.52 on our serverless compute
requires-python = ">=3.10" - we are using python 3.11.10

[FristrisX]

Fix:
Used to use Builtools to build the wheel file for our custom implementation of DQX.
We now use Poetry to build our implementation of DQX and the problems are now gone