question

jbakuwel avatar image
jbakuwel asked

Local MQTT updates vs MQTT updates to VRM

Hi all,

According to KeepAlive paragraph in https://github.com/victronenergy/dbus-mqtt/blob/master/README.md I should run something like this:

while :; do mosquitto_pub  -m '' -t 'R/e0ff50a097c0/system/0/Serial'; sleep 5; done

in order to be able to maintain active subscriptions with the MQTT broker on Venus for clients on the local LAN.

The MQTT broker on Venus has this timeout functionality in order to prevent overloading VRM (fair enough). But ... would Ii not start to update VRM too frequently if I implement this keep alive function?

Instead of having to run a funny out-of-bounds script, would it not be better to be able to set a "keep alive" flag that the MQTT broker can check for use cases where local clients use the MQTT broker on Venus and would like to continue to be informed about changing data?

regards,
Jan



Venus OSMQTT
1 comment
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

woogieboogie avatar image woogieboogie commented ·

Hi @jbakuwel, @ben, interesting thread.

I have a remote python script that's reading values from the VRM ok but suffers from the keepalive issue. For now I have to separately run the VRM remote console to keep the broker awake and then I get my updates ok. Otherwise I don't.

I've seen the "solution" from the manual is to run mosquito_sub script, but it's hard going if (like me) you've not used mosquitto before. I can see Mosquitto is installed on the Venus but can't see a mosquitto bridge file and have no idea how I should change it anyway (to keep the MQTT broker alive). Any pointers would be much appreciated.

Finally (and separately) any idea how the VRM remote console manages to constantly get updates? It must circumvent the keepalive issue somehow??

Thanks,

Cliff


0 Likes 0 ·
3 Answers
mvader (Victron Energy) avatar image
mvader (Victron Energy) answered ·

Hi Jan, I dont think that would work.


The timeout handling is in the service that publishes the data on mqtt. Its not in any of the brokers themselves.



2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

ben avatar image
ben answered ·

The motivation for the keep-alive is to prevent the Victron cloud MQTT server (which is not the same as VRM, although it shares the same authentication credentials) from receiving unnecessary MQTT traffic from Venus instances where MQTT is turned on but not being used by any consumers.

It is true that, for the small subset of users who are using MQTT -- either over the LAN or by connecting to the Victron cloud server -- this creates traffic for Victron's cloud server.

It is also true that some subset of those users may not care about the cloud server, and so for those users there is bandwidth being wasted sending MQTT broadcast copies to the cloud (and processing time and space being wasted on Victron's cloud server itself).

If you are concerned about this, you can reconfigure the broker settings on your local Venus device such that it does not propagate copies to the cloud server. Presumably, Victron is not (yet) concerned about this situation. My guess is that that is because (a) MQTT is generally pretty lightweight and (b) there are very few Venus clients actually using MQTT, so far.

3 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ commented ·

Disabling forwarding to cloud broker, and more changes, are coming:


https://github.com/victronenergy/venus/issues/480

0 Likes 0 ·
ben avatar image ben ♦ mvader (Victron Energy) ♦♦ commented ·

Excellent, yay!

0 Likes 0 ·
gainestr avatar image gainestr mvader (Victron Energy) ♦♦ commented ·

Has this been implemented yet and if so, how is this done. I thought I could simply disable MQTT on LAN (SSL) but it also disables my local (plaintext) connection as well.

Edit: Enabling/Disabling VRM two way communication was the answer to my question.

0 Likes 0 ·
jbakuwel avatar image
jbakuwel answered ·

Hi Ben & Matthijs,

Thanks for your replies, very helpful.

I think we should all care about not overloading the Victron cloud server.

I've created a clone of the dbus-mqtt service (I called it dbus-mqtt-local) that continuously updates a local broker. This way I can keep VRM, not overload the Victron cloud server and also have the local service I need.

cheers!
Jan


4 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

ben avatar image ben ♦ commented ·

Hey, great idea with the separate service.

Just a few points of clarification:

1) Let Victron worry about the loading of the cloud server; anything else is premature optimization on our part, especially we can't see their side's scale or economics anyway. The data payloads are delta-based and very small anyway, so I doubt this is at all an issue today. (Optimizing your own data uplink bandwidth is an entirely different story!)

2) While creating another service is creative, it's pretty trivial to edit the mosquitto bridge config file.

3) Finally, in today's architecture, the MQTT and VRM service are fed differently. (Please inform me if you found otherwise.) That is going to change one day soon, I believe.

Cheers!

0 Likes 0 ·
jbakuwel avatar image jbakuwel ben ♦ commented ·

Hey Ben,

Re 1) While I see your point of the data payloads being small, in my opinion (having worked in the IT industry for a couple of decades), it's just good practice not to do something that is not necessary. And as you say, the knife cuts both ways, it's good to not send data that doesn't need to be send although it would not affect my link in a noticeable way.

Re 2) I don't think I can edit the mosquitto bridge config file for my purpose or am I missing something here? I do want to update local clients as data changes (ie. very frequently) but do not want to update the Victron cloud server with the same frequency. Hence using another service. Which is trivial do to (too) :-)

Re 3) I must say I haven't spend much time trying to understand what is what on the Victron cloud end. I'm dealing with stuff at my end and will happily leave dealing with the remote end to the capable masters at Victron.

cheers!
Jan

0 Likes 0 ·
ben avatar image ben ♦ jbakuwel commented ·

Hey,

Regarding (1), I'm just suggesting that hacking up VenusOS just to try to optimize away something that Victron is absorbing the cost of doesn't pencil out very well for us end users. If you also have another reason to do it anyway, that's a little different.

(2a): yes, you can edit the bridge config. You have root on your device. :)

(2b): to be clear, the keepalive timeout doesn't have anything to do with the data update "frequency" or data volume. You could send keepalives every second and you'll still get the same number of MQTT messages transmitted. This makes me wonder if you are doing something different than I (or you) thought you were, or if I am misunderstanding what you are want to achieve.

(3): I completely agree with your last sentence, which is why I wrote point (1). :-)

Cheers.


0 Likes 0 ·
stoeff99 avatar image stoeff99 ben ♦ commented ·

Hi Ben,

Two questions:

1) I've been trying to figure out where the bridge config file is. Is it the vrm_bridge.conf file in /data/conf/mosquitto.d or is it in another place?

2) what do I need to edit within the conf file to stop the data being sent to the cloud server and that I only have mqtt enabled locally?

Thanks

Chris

0 Likes 0 ·

Related Resources

Victron MQTT readme

Victron Venus OS Open Source intro page

Venus OS GitHub (please do not post to this)