Instantaneous Demand not calculated correctly

[danstreeter]

The values provided over MQTT for Instantaneous Demand (Node 0702.04.00 in payload) is returned as a signed 24 bit integer in hex... (see https://zigbeealliance.org/wp-content/uploads/2019/12/docs-07-5356-19-0zse-zigbee-smart-energy-profile-specification.pdf page 248, table on line 6028, first row) when this value is represented as a negative value (I am generating more than I am consuming) the current function in https://github.com/unlobito/ha-hildebrandglow/blob/mqtt/custom_components/hildebrandglow/mqttpayload.py#L160 calculates this incorrectly and you get a widly high value intead of a negative value.

Scenario
I'm curently feeding the grid with 785 watts - so the Glow app reads -785, the values coming in from the data source via MQTT is FFFFFCEF, which is being calculated as an unsigned int as 4294966511 (Clearly not what i'm using! haha)

The fix for this is to implement a two's compliment conversion of this number to get the signed integer out the back.

Forgive me, my mathmatical knowledge on this is 'literally learnt about it this morning' - so i'm probably saying it all wrong.

But
hex FFFFFCEF == binary 11111111111111111111110011101111 == -785 decimal from signed 2's complement.

I think this SO link may provide some ways to do it, but will need testing obviously: https://stackoverflow.com/a/6728042/5889983

Cant do it myself as on work time until this 1730 GMT+1, but could possibly have a look this evening when off work.

[danstreeter]

Useful reference document where someone has mapped the MASSIVE zigbee energy protocol spec to the fields we need (about 2 comments down): https://forum.glowmarkt.com/index.php?p=/discussion/36/anyone-able-to-develop-a-home-assistant-custom-component-or-templating

[danstreeter]

WIP fix implemented on my live system at home - will allow it to run and give it a test tomorrow when the sun is back out generating and consumption is lower than the evening.

_Tried to mock out and falsify the inbound data streams, but there is too much to do at once, and data streams from PVOutput as well as MQTT to mock - so gave up! 🤣 _

Initial tests of new function are ok, and tests changing the MQTT server to my own, and sending it negative values for the 0702.04.00 value now calculate correcly and appear as they should - so I'm reasonably confident - just want to check that once all data feeds are coming in as they should in prod - it all still works ok.

If all works as I think it should tomorrow - ill raise a PR

[rit001]

Was anyone able to resolve this? I've just started to play with HA as I have a Glow interface and solar, so my ha console is currently reporting that my energy usage is around the same as a large chemical plant :)

[Daibutt]

Was anyone able to resolve this? I've just started to play with HA as I have a Glow interface and solar, so my ha console is currently reporting that my energy usage is around the same as a large chemical plant :)

Yes I made another sensor like this;

sensor.electricity_consumption

• platform: template
  sensors:
  current_grid_import_smart_meter:
  friendly_name: "Current Grid Import (Smart Meter)"
  unit_of_measurement: 'Watts'
  value_template: >-
  {% if states('sensor.electricity_consumption') | int > 30000 %} {{ states("sensor.electricity_consumption")|int - (2**24) }}
  {% else %} {{ states("sensor.electricity_consumption") }} {% endif %}

Seems to work OK so far.

[danstreeter]

My change has been running ok for quite some time now (mentioned in #32 (comment)) - I'll have to pickup where I got with it and raise a PR etc if it works etc...

Update - Ok, so i have the fix committed to my fork, upstream has gone ahead since I did it so need to rebase, then request a PR. I'll try and do this tonight has have a busy day today.

Update - Took two seconds, PR raised as seen in comments below.

[rit001]

Thanks for the replies - so much of this seems to be based around vendors creating solutions and then leaving it up to those with skills in the 'community' to make the solutions work/be useful.

So thanks again.

[danstreeter]

creating solutions and then leaving it up to those with skills in the 'community' to make the solutions work/be useful.

Thats the beauty of open source software! Someone comes up with an idea they may or may not have the expertise to solutionise, and the community come together and go - yeah, we can make that work!

Anyway, my PR failed some sort of pipeline merge, not sure why - cant see any details. I've asked for any info.

[rit001]

Now that is something I can help with, more detail is provided via the Travis CI link that is provided at the bottom of the Build Failed report. It takes you to

  https://app.travis-ci.com/github/unlobito/ha-hildebrandglow/builds/239548384

[danstreeter]

Ahar. Thanks. I’ll take a look tomorrow.

[Daibutt]

Agree Dan, I've learnt so much from just reading all the comments. I am a total noob in HA and with MQTT. I just followed what all the smart people on here have documented. Thank you guys!
Not sure if this helps or is relevant.... my meter is SMETS1 and I was getting much lower "massive" numbers when I was exporting than some people reported.
I'm not sure if SMETS1 numbers are different format than SMETS2 but the 24bit conversion worked for me (-224). If not maybe try -232?

[Daibutt]

Hmm that obviously didn't work... should read 2^24 and 2^32

[rit001]

Working from the original document

 https://zigbeealliance.org/wp-content/uploads/2019/12/docs-07-5356-19-0zse-zigbee-smart-energy-profile-specification.pdf

Attribute InstantaneousDemand is described as

ID 0x4000 which I guess maps to 40 00 in the naming structure
Type Signed 24-bit Integer
Range -8,388,607 to 8,388,607

So the maths should be based around 2^24, unless the number ends up being modified by the MQTT gateway by accident/design.

{edited to note that the MQTT gateway may change the expect value in some way}

[danstreeter]

Ok 'fix' pushed and is going through tests in Travis now - hillariously (and stupidly) the change was obvious, and actually present in my local running version of home assistant, just not backported to my repository where I commited from. Doh!
Oh to rush eh!
Update - More changes added to comments that were picked up during Travis run, but not in local run of same command.

[townsmcp]

@rit001 and @Daibutt @danstreeter are the issues you have all seen here related to the issue I have raised in #47 do you think? Lol, I know my house also is not using the same elec from a large chemical plant 😂. I don’t have solar though so am not feeding electric back to the grid