question

ozdw avatar image
ozdw asked

CCGX MQTT issue - no activity

A couple of days ago, my nodered flows (on a linux box) subscribed to mqtt.victronenergy.com stopped working. Attempting to diagnose the issue, I used

mosquitto_sub -v -I myclient -t '#' -h ccgx

against the CCGX locally and receive no updates and the only value returned is for the topic "N/my_identifier/system/0/Serial". Previously, this would've returned a large volume of data.

I eventually found how to set a root password, then logged into the CCGX as root and looked at the mosquitto log files and find nothing unusual. Looking at the dbus-mqtt log, I don't see anything suspicious.

I've rebooted the CCGX and the issue persists. It's running the latest firmware.

I would appreciate any guidance on this puzzling issue.

MQTT
1 comment
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

ozdw avatar image ozdw commented ·

I should add that everything in the system is working normally and a modbusTCP client app shows expected data updates are happening on modbus. It's just MQTT that's apparently not being updated.

0 Likes 0 ·
4 Answers
ozdw avatar image
ozdw answered ·

Interesting. After three or four days and nothing changing here, it's working again. I wish I knew what happened.

2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

tabbertmj avatar image
tabbertmj answered ·

I'm currently seeing the same thing. It was working for me a week or two ago, now I only get the serial number. A reboot does not seem to fix it.

2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

tabbertmj avatar image
tabbertmj answered ·

I updated to RC v2.62~10 and still do not have data.

OK, once I published to the below about 7-8 times it finally returned data. The first few times it only returned the serial number. There seems to be a flaw.

R/{siteID}/system/0/Serial
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

mvader (Victron Energy) avatar image
mvader (Victron Energy) answered ·

Hi,


Try

1) the mfd app: open http://venus.local/app/ in a browser, if that works, mqtt works. If it only loads the page, but then no data or even an error message abour MQTT, then there is an issue with the MQTT settings.


2) similarly, go to VRM, go to the dashboard od your site, if it says Realtime, then MQTT works.


We’d be getting tons of questions if MQTT stopped working, as both above much used features rely on it.

Most likely there is some issue in your handling of sending the keep alives.


hope that helps!

7 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ commented ·

Ps. Make sure to also read this: https://github.com/victronenergy/dbus-mqtt#determining-the-broker-url-for-a-given-installation


most likely that will solve the issue

0 Likes 0 ·
tabbertmj avatar image tabbertmj commented ·

mfd.pngHI Matthijs, Today, the MFD page is not loading. It did load yesterday with V2.60. It does, however start populating data in MQTT. This may explain why I was seeing data last week, but not this week. I probably had the page loaded on one of my tabs.

The VRM portal does say Realtime, but does not populate the MQTT data like the MFP page.

Last night, in order to get the data to populate, I had to publish to "R/{siteID}/system/0/Serial " more than once in order to get data to start populating. I would expect that I only need to publish once every 50-60 seconds to keep it alive.

Thanks,
Mike

0 Likes 0 ·
mfd.png (52.9 KiB)
mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ tabbertmj commented ·

Hi @tabbertmj, I found your site in VRM (Montana, right?). The issue is that the CPU is overloaded. If you look at the Diagnostics page on VRM you’ll see that the load average was 4; also it shows a watchdog reboot.


and while browsing on VRM you see “installation too busy, at the lower right bottom of the screen).


5 VE.Direct devices, as you have, is really the max, and then enabling modbustcp adds significant load (even if you’re not querying it) and similarly MQTT.


the solution would be to either reduce the number of connected devices (I understand ofcourse that thats probably not really an option) or disable a few features, or replace by a Cerbo as that has more CPU power.

The other thing that helps is to se the graphical display to the menu pages, to not have the overview page with the “moving ants”; that page also causes quite the cpu load.


ps. while I watched, a saw load average of 4 and 5; and it was running v2.60

0 Likes 0 ·
tabbertmj avatar image tabbertmj mvader (Victron Energy) ♦♦ commented ·

Yes, Montana is the correct site.
I'm not sure where the Diagnostics page is on the VRM. There were several reboots this morning which were resolved when I rolled back to V2.60.
The “installation too busy" message has always displayed for my site from day one when I had two VE.Direct MPPT's and a BMV. So that has never been a red flag. I'll watch for that in the future to see if the behavior changes.

If MQTT is always running for the VRM and internal pages, how is there that much extra load by using it? It seems that if the GX device is already using it, it can't be disabled. I can try to switch over my integration from modbudtcp to MQTT. Upgrading to a Cerbo is not feasible currently. Hopefully that can be an option by the end of the year. Until then, I will try to move to MQTT and disable modbustcp.


Thanks,
Mike

0 Likes 0 ·
mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ tabbertmj commented ·

Hi, ok I hope that works for you. I’ll send you your diagnostics page in a private message, but basically you need to replace whatever is at the end of the url by /diagnostics

0 Likes 0 ·
mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ mvader (Victron Energy) ♦♦ commented ·

Ps. Also, mqtt.victronenergy.com/ Is for 100% sure the wrong broker url to subscribe to topics. See the link I posted above

0 Likes 0 ·
tabbertmj avatar image tabbertmj mvader (Victron Energy) ♦♦ commented ·

I found the diagnostics page, thank you.
I m not using mqtt.victronenergy.com, so I don't know where that is from. I connected with MQTT Explorer using my local IP address.

What I find interesting is that I have not published to R/{siteID}/system/0/Serial in over 24 hours, but the broker is still populating data. This seem to go against the 60 seconds shown in the documentation, and what I was seeing earlier in the week. What would cause the change in behavior?

"Default keep-alive interval is 60 seconds. If the CCGX does not receive any read or write requests during that interval, the notifications will be stopped, until the next read or write request is received. So to keep the notifications running, you'll have to send a read request regularly, for example: "

0 Likes 0 ·

Related Resources

Additional resources still need to be added for this topic