question

hobohome avatar image
hobohome asked

Venus GX / Smart Solar feature Suggestions

Hi, I'm not sure if this is the correct forum for this - I'm sure somebody will let me know if it is not. :-)

In my off-grid home, I use the open source project OpenHAB to manage energy consumption, display status information and produce warnings for potential issues.


First suggestion - No MQTT shutdown

I connect to the MQTT broker on my venus GX to obtain a the huge wealth of data that the Victron systems make available.

While I fully understand that Victron do not want every Venus user in the world permanently connected to their MQTT broker (the Venus OS mirrors data from the local broker to the Victron broker), resolving this by shutting down the Venus data propagation every 60 seconds seems like a bad idea (to me anyway). I have of course prevented this shut-down on my system (by publishing to the broker every 30 seconds), but in so doing, I am generating traffic out of my network AND placing an additional load on the Victron broker.

Perhaps it would be a better plan to have a script wake up every 15 minutes to publish the data from the local broker to the Victron broker, then close down THAT link (only)? This would mean that a keep-alive would no longer be required, the local broker would continue to get updated, Victron brokers would not be getting hit as much and (very importantly) retained data on the local server would no longer disparate if the keep-alive stops.


Second suggestion - MPPT PMax data

My automation system tries to optimise energy usage by attempting to use energy when there is an excess (in the middle of a sunny day) and conserve when there is a shortage (overcast).

One thing that makes this difficult, is that once the solar controllers go out of bulk mode, I never really know how much more energy I can use before I start discharging the battery (or making the controllers drop back to bulk). The data value that would make this really easy would be "estimated max power" - the controllers estimation of how much output power would be generated if the controller went back to MPPT mode.

Right now I estimate this data by combining information from my own weather station (solar radiation) with information from the Venus controller. This tends to be fairly unreliable due to the infrequent updates from the weather station (20 seconds) and the fact that I have three solar arrays pointing in three different directions.

I realise that the MQTT part of the Venus is open source and I would love to contribute, but while I have some programming experience, none is with Python and I don't think my skills would be up to producing anything worthwhile. I would of course be very happy to help with testing etc.

Venus OS
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

2 Answers
ben avatar image
ben answered ·

Regarding the MQTT configuration, I have a few thoughts as an end user/casual hacker:

1) Since almost no one is using MQTT today, Victron’s approach is okay from their servers’ point of view. It’s probably substantially less than ideal when more people use it, as you suggest. But I doubt they are keen to pre-optimize for the future at this point; there are a lot of other things that I think have to be higher priority.

2) Nevertheless, it’s relatively straightforward to reconfigure the mosquitto server broker, bridge, timeout, and so on. You can do this yourself on your Venus device if you go dig around in it. Note that you have to take extra steps to ensure your changes survive an OS update, or you can re-apply them manually after each upgrade.

3) Despite your idea to reduce the frequency of pushes to the cloud server, note that any downstream consumer may be depending on frequent updates as a way to obtain timestamping. The Victron-published MQTT messages do not currently include a timestamp, I don’t believe, and MQTT headers don’t capture this either. So bunching up a series of updates and pushing it all at once would throw away some timing information. Venus drivers would need to be rearchitected to include that timing data as part of the payload... a substantial effort given how many publishers (“drivers”) there are now.

4) Despite my concerns about #3, I think there is one scenario where it makes sense to pursue #2 as an individual: if we are not using the cloud server for any reason, and if our bandwidth is constrained or expensive, and if we are going to otherwise resort to keep-alive to maintain continuous monitoring on our LAN, then... I think it makes sense to go in, disable the server timeout, and turn off the cloud bridge.

Whew!

7 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

hobohome avatar image hobohome commented ·

@ben thanks for your comments - its good to hear other viewpoints.

I totally disagree that MQTT is not commonly used. It is the protocol of choice for all forms of home automation including the current market share holders, Home Assistant and OpenHAB. It is very light-weight and almost perfect for making all types of data available. Recent advances in auto discovery and standardisation (see the homie standard) will also have a big impact on the growing use of MQTT.

#2 - I agree that it is fairly simple to alter the scripts that control the current broker linking, my suggestion being that it be done ONCE for the benefit of all. I don't have the python skills unfortunately.

Continued .... ->

0 Likes 0 ·
ben avatar image ben ♦ hobohome commented ·

You misunderstood me. When I said it's not commonly used, I mean that very few Victron users are using MQTT. It's just a nearly-hidden, power-user feature. Sorry for the confusion.

0 Likes 0 ·
ben avatar image ben ♦ hobohome commented ·

Reconfiguring mosquitto does not require any python coding. You just edit the configuration file. (Are you talking about something else instead?)

0 Likes 0 ·
hobohome avatar image hobohome commented ·

#3 - I could be wrong about this (please correct me if I am) - my understanding this that the reason for the timeout shutting down the link between D-Bus and MQTT on the Venus device, was to prevent continuous traffic to the Victron MQTT broker. If this is the case, the flow of data is currently NOT continuous. The only problem with the current setup is the way they implemented the shutdown - by stopping publishing the data to BOTH the local broker AND the remote broker. I'm only suggesting that it would be more useful to ONLY stop the link to the remote broker. This would leave the local broker functional to serve local needs.


BTW - whats with the 1000 char limit on comments?

0 Likes 0 ·
ben avatar image ben ♦ hobohome commented ·

I don't know Victron's rationale, but your guess seems plausible.

When you suggested changing it to only connect every 15 minutes, I assumed you were envisioning the updates being queued for delivery every 15 minutes. This creates roughly the same bandwidth egress and throws away the timing data, which would probably not be what you'd want.

Your new suggestion here, to simply close the connection to mqtt.victronenergy.com when the keep-alives stop, seems reasonable. You could also just completely disable it from the get-go and nothing bad would happen... as far as I know, today, nothing else depends on the cloud broker.

0 Likes 0 ·
ben avatar image ben ♦ hobohome commented ·

Yes, the 1000-char limit is obnoxious. Also, this Q&A software is ill-suited for more complex discussion. It gets too hard to tell what we're talking about, which is why I try to number my statements so at least there's some crude way to reference them!

You could take this particular conversation to the dev-venus group and at least there we would be able to use regular quoting and threading tools to converse. Also, a lot of other hackers are over there.

0 Likes 0 ·
mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ hobohome commented ·

Hi, yes making the keepalive work only on the connection to the cloud broker seems a good idea to me too.

Some issues will be:

- cpu load. Dbus-mqtt is already far too cpu hungry; see issue we have on Venus. And

- developer time. As Ben said: we can only do so much

The good news though is that after being dormant for a while; its coming back onto the picture again slowly (hence the activity on that issue). For some nice features like real time updates when browsing VRM on your browser and also adding more remote monitoring but primarily configuration access to VictronConnect in combination with Venus.

0 Likes 0 ·
ben avatar image
ben answered ·

Regarding publishing new values to MQTT, that part is quite easy to do. What it seems might be hard is actually convincing the MPPT controller to expose that “estimated max power” value. Do they calculate that today? Or is it something we could somehow infer from the data we do have streaming out of them?

4 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

hobohome avatar image hobohome commented ·

Yep - the issue is getting Victron to expose that value. It may be that the controllers don't even calculate that value when they are not in tracking mode.

The documentation for the ModBus data available from Morningstar's MPPT controllers says that they make available a value called duty-cycle (being the PWM value required to maintain the current target voltage) when the controllers have moved beyond the tracking phase. Sadly they wrote it in the documentation but forgot to actually implement it! :-(

0 Likes 0 ·
ben avatar image ben ♦ hobohome commented ·

Maybe you could infer it by putting an inductive pickup on the outside of the controller and measuring the frequency. :)

0 Likes 0 ·
mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ ben ♦ commented ·

Hey both, the mppt doesn’t calculate it.

Even ‘worse’: when not in tracking mode; its impossible for it to know the available excess power. It could try and look it up; but that would mean going to maximum power point: also putting out all the power (where else could it go?) which means the battery would get a quick energy boost; and relating voltage spike.

The only feasible way that I can think of is activey controlling the dumpload with a closed loop (turn it up and down based on voltage / mpp mode; keeping the system in balance); or having a irradiation meter at the side; which would be (far) more straight forward I’d think.

0 Likes 0 ·
hobohome avatar image hobohome mvader (Victron Energy) ♦♦ commented ·

Yea, I thought as much. However, when the MPPT is not tracking, I assume it is using PWM to control the output current. Making the duty cycle value available as a percentage would give a fair approximation of what percentage of available energy was being allowed to flow.

I do have an irradiation meter in place now, but as I have 3 MPPTs controlling 3 arrays pointing in different directions, the data it produces is very inaccurate. I guess I need one per array. I prefer the first option :)

0 Likes 0 ·

Related Resources

Victron Venus OS Open Source intro page

Venus OS GitHub (please do not post to this)