question

jago avatar image
jago asked

MQTT values periodically corrupted (2.9X and 3.00beta)

Hello - I am using a Multiplus GX with a pylontech battery stack.

I am collection the process data via the internal mqtt-broker from the gx unit - this values are used in different systems (influx, nodered, grafana, home assistant). My problem is, that the battery values (current, voltage, power) are sometimes corrupted (like a overflow). First I thought, that my data processing is doing "somthing wrong" - but I have broken it down to the internal broker and i can see (e.g. with mqtt-explorer) that the .json data from the multiplus seems to be wrong...

Attached you can see a screenshot of the described phenomenon.

vic-mp2-mqtterror-current-01.png

It happens first with the 2.9X version of the venusOS - actually I have installed the 3.00-21 as beta to see if the behavior changes - it does not change.

I have although activated the VRM-Data (which was deactivated in the past) to see if I have the same strange data - no: the data in the VRM seems to be ok. I haven never ever get some warnings or alerts from the system, so I thing the internal values "inside" the multiplus are ok - but the mqtt data is definitely corrupted.

Perhaps an interesting bug - If anyone has an idea, what to do or to check: feel free to comment my question ;)

Multiplus-IINode-REDMQTThome assistant
3 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

hominidae avatar image hominidae commented ·

I only started with victron GX on v2.91 and I've never seen it different from this....these are float numbers/values, so why care...the subscriber can take care of rounding/marshalling these, if need be.

I've never had a problem using these values in my Node-red flows and calculations.

0 Likes 0 ·
jago avatar image jago hominidae commented ·

thanks for your response - if you take a look at the screenshot (History data) you can see, that the reported payload is {"value": -1996.800... } - it is not a mater of rounding, these values are physically wrong... There has never ever been a current of 2000 Amps ;)

And: there is no alert or warning tripped, so in my opinion, internally the values are ok, but there is a bug in the reporting module publishing the mqtt data to the broker

0 Likes 0 ·
hominidae avatar image hominidae jago commented ·
Ah, yes...didn't catch that on your screenshot.

That's a valid point, then.

However, how frequent are these "glitches" ?

I am on v2.92 on my CerboGX and have never seen such alternating/inkonsistent numbers.

I am logging these into my own local influxDB as they come in and again, no glitches yet since last 5 month uptime, when the system has been first installed.

0 Likes 0 ·
8 Answers
jago avatar image
jago answered ·

thanks for your response - if you take a look at the screenshot (History data) you can see, that the reported payload is {"value": -1996.800... } - it is not a mater of rounding, these values are physically wrong... There has never ever been a current of 2000 Amps ;)

49788-vic-mp2-mqtterror-current-01-detail.png

And: there is no alert or warning tripped, so in my opinion, internally the values are ok, but there is a bug in the reporting module publishing the mqtt data to the broker


3 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

hominidae avatar image hominidae commented ·
as alreaday said above, I did not see/catch these kind of off/odd-values in my system. I am logging Battery Amps locally in my influxDB and only see values that are inside the valid range of my system setup (running a Cerbox GX with v2.92)

How frequent are you seeing these glitches?

0 Likes 0 ·
jago avatar image jago hominidae commented ·

I would guess one or two times a day...

I have checked the can-bus error counter: no lost packages.

mp2gx-can-pylontech-noerror.png

I have removed the termination resistor - no change

0 Likes 0 ·
hominidae avatar image hominidae jago commented ·

strange.

This is from my system, from over the last 7 days.

No glitches...all well within possible real value ranges.

1679486226130.png

0 Likes 0 ·
1679486226130.png (169.7 KiB)
jago avatar image
jago answered ·

can you check the firmware version of the pylontech batteries?

I have added one new US2000C block to my system and probably this could be an issue...

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

hominidae avatar image
hominidae answered ·

no, I can't...it is said, that in case of an upgrade/change, the newest module (highest S/N and therefore probably the newest FW) must be made master in the stack.

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

jago avatar image
jago answered ·

I have checked the firmware versions of the US2000C - every module is up to date (Jan 2023 and Sep 2022 - based on the hardware revision all modules have the latest possible versions: US_C_Nt_V1.7 and US_C_V2.8)

The system operates flawless - there are no errors (temp or over current) logged and the data logged to the VRM-Portal has no glitches - to me, this behavior can only be explained by an internal software bug of the GX module...

I am not alone ;)

I have found a post with similar "glitches" on pylontech batteries:

https://community.victronenergy.com/questions/54257/pylontech-und-ccgx.html?childToView=195792#answer-195792

But: read the content carefully: it is not a matter of linking the batteries with the correct cabling or having a poor CANbus quality - be aware, that the overall performance of the running system is OK: no errors, no faults, no current or temperature warnings, no logs - just wrong values communicated by the GX module (it does not matter if you use mqtt or modbus)...

I hope someone with connections to victron can take a look to this - or give us a feedback how to debug the internal data.

kind regards ;)

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

mvader (Victron Energy) avatar image
mvader (Victron Energy) answered ·

Hi, just a note that I’m following this conversation. I am from Victron.

But I have no idea where the bug would be.

The code that takes data from our internal databus (D-Bus); and publishes that to MQTT is here:

https://github.com/victronenergy/dbus-mqtt">https://github.com/victronenergy/dbus-mqtt

Fairly simple code.


I don’t remember ever seen such weird values for DC current in VRM.

Is it always the current that has an issue?

Can you show screenshots of graphs that show how often it happens?

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

jago avatar image
jago answered ·

Hi, I have checked my DBs and I was surprised, that my retention policy was set to 7days only on my influxV2 instance - all the data are "collected" by telegraf directly from the internal mqtt broker of the victronGX module. It looks like this:

20230329-victronbroker-512-dc-7d-values.png

as you can see: all the values coming from the topic "N/XXX/battery/512/Dc/0/..." seems to be temporarily corrupted - the green SoC seems to be ok, but:

.../Current ...Voltage .../Temperature .../Power

have the same glitches.


2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

jago avatar image
jago answered ·

I compared the values from .../Voltage and the CellMin and CellMax voltage values from the N/XXX/battery/512/System/MaxCellVoltage and ...MinCellVoltage and copied following screenshot:

20230329-victronbroker-512-volt-7d-values.png

The database of my home assistant server although stores the values from the GX-Broker directly:

20230329-haos-march2023-values.png


thanks for keeping an eye on this issue, kind regards ;)


2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Stefan Boldt avatar image
Stefan Boldt answered ·

I am running firmware 3.30 as large version and I have the same glitches all over the mqtt values. Here an example:

Value is reported as 1.083,3 kWh and suddenly it goes to 915,03 kWh and back to normal.

1711442606816.png


1711442658647.png

I have the suspicion that it has to do with restarting HA (not reboot). Almost everytime I restart HA, my Node-Red instance on the Cerbo (not the HA Addon) crashes and restarts again. Not sure if this two problems are somehow related.


1711442606816.png (43.4 KiB)
1711442658647.png (42.6 KiB)
1 comment
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

hominidae avatar image hominidae commented ·
...are you running an mqtt-bridge of some sort between the GX and HA or another external broker?

Looks like a glitch with retained values, when the HA side restarts, probably?

0 Likes 0 ·