question

(Please note: I came up with the solution near the bottom of this reply...as I wrote this entire post.)

When I have the http://venus.local/app/ open, immediately it tells me "Your Dashboard is Empty. Make sure your services are properly connected."

Everything is enabled as it's supposed to. After a few seconds, data will start coming in to the dashboard.

I didn't even know about the venus.local/app until today. I've connected to the IP straight up which mirrors the control panel nicely), shown here, plenty of times.

However, the Venus web app is doing something else. When I watch the mqtt topics with MQTT-Explorer (great app, by the way), the topics simply do not exist when I connect. But if I start up the Venus app, when IT starts receiving data, so does MQTT-Explorer, but not until.

Before I start the Venus app web page, showing MQTT-Explorer:

After I start the Venus app pages, after a few seconds, the MQTT-Explorer lights up with the expected topics:

After I close the Venus app web page, after a bit they go away again:

In the middle screenshot, notice the >R topic at the bottom:

That was added by the Venus app webpage code, somehow. I can't see anything in the Chrome headers for the websocket connection that seem overly obvious to me, but I'm not very familiar with websockets. Here is the entire JSON for that keepalive key:

[
  "N/+/system/0/Serial",
  "settings/0/Settings/System/VncLocal",
  "system/0/Dc/Battery/Voltage",
  "system/0/Dc/Battery/Current",
  "system/0/Dc/System/Power",
  "system/0/Batteries",
  "system/0/Dc/Pv/Power",
  "system/0/Dc/Pv/Current",
  "charger/+/DeviceInstance",
  "inverter/+/DeviceInstance",
  "vebus/+/DeviceInstance",
  ...plus 50 more lines I removed for brevity
]

While the Venus app is connected, the primary topic values update frequently, about every 5 seconds it seems. I'm specifically watching key "N/e8eb11e512f4/system/0/Batteries" because it is a JSON object with several values in it.

After I've closed the Venus app page, and disconnect/reconnect MQTT-Explorer (which refreshes the available topics), that keepalive key is no longer visible.

Connecting directly to the mqtt server in the Victron setup (mosquitto version 1.6.10), or connecting to my own mosquitto server (mosquitto version 2.0.18) and utilizing a bridged connection -- neither show the topics I'm looking for.

SUCCESSFUL RESULTS:

It's quite obvious that the giant array above contains topics. I grabbed the system/0/batteries value and did a publish of:

R/e8eb11e512f4/keepalive with a value of ["system/0/Batteries"]

This IMMEDIATELY causes that SINGLE item to appear and refresh every 5 seconds, both in the direct connection and in the bridged connection:

This is my own MQTT server, showing the bridge connection receiving the updates.

After about 1 minute, the Batteries topic disappears. It seems to me that this is some sort of power-saving feature, reducing CPU overhead by not pushing data from the source to the mqtt server topics. Unless someone is actively seeking it. This means that a simple bridged connection won't pull new data, because it's not there. Any app or dashboard will have to republish that keepalive message to ensure that the keys you seek are published. Interestingly enough, I was able to publish the same message to the bridged mqtt broker and it is transferred across, but only because of my mosquitto.conf for that connection, using the "both" keyword.

# -----------------------------------------------------------------------------
# Bring in topics from other MQTT brokers (in 0, 0 represents QoS)
# -----------------------------------------------------------------------------
# Bridge Victron Topics to DWO Dragon Mosquitto
connection dwo-to-victron
address 192.168.60.16:1883
keepalive_interval 10
start_type automatic
remote_clientid dwo-dragon-mosquitto
try_private false
cleansession true
topic N/e8eb11e512f4/system/0/Batteries both 0
topic N/e8eb11e512f4/system/0/Serial both 0
topic R/e8eb11e512f4/keepalive both 0

I hope all this information helps someone who stumbles across this. Basically, tl;dr is you have to push a keepalive to the server every (less than 1 minute) to keep the topics you seek continuing to be published.

Please note: I'm NOT any kind of mqtt expert. In fact, other than looking at all this about two years ago, going "poke, poke, huh??" and then the last two days reading up on mqtt and trying to get a bedtter grip on it, then running into this problem, and TEARING it all apart and EXPERIMENTING, that's all I know.

Maybe the keepalive is a standards or protocol thing, but I suspect not. I more suspect this is what Victron is looking for under the hood to save power and CPU cycles. (They are a "power company", you know.) Earlier today, I connected to the testing server (aka "you shouldn't use it for anything production or expect consistent results and it's all publicly visible") at test.mosquitto.org and I saw thousands of topics being updated, without my pushing any data to it whatsoever.

What I'd like to know is if there is a setting, a more "passive", or "connection-based" way to do this without requiring some trick to push a keepalive every minute to get the desired keys to keep updating. Right now, I'm thinking some form of "healthcheck" is going to be the trick.

@flovic @CaptainPalapa

Thanks, really helpful.

Moving this to the modifications space as it belongs there.