question

jacola avatar image
jacola asked

ESS 3 phase 3 with external control, one phase sometimes ignores setpoint

I might repeat myself but i am scratching my head and the problem is not solved.
My ESS is 3 phase, with Multiplus-II 5000 on firmware 474, Venus on 2.60. I use external control.

Sometimes one of the Multiplus units on one phase apparently decides to ignore the setpoints and starts doing odd things .. with the result being a wild ride between feed-in and consumption.

The situation resolves after some time, but that can take an hour or so. The other multiplus'es follow the instructions without issue.

this is the problematic phase only, SP is setpoint.
The overview of all 3 phases for a longer interval is here:
you can see that normally the inverter follows the instructions well ..


Any advice appreciated.

Johannes


Multiplus-IIESS
screenshot-2020-09-13-at-125742.png (229.2 KiB)
screenshot-2020-09-13-at-124726.png (740.1 KiB)
1 comment
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

jacola avatar image jacola commented ·

here we are again, producing 2500W instead of charging 400 .. not funny .. this is only one phase, the others show similar behavior. out of the blue for me ..

0 Likes 0 ·
screenshot-2020-09-15-at-084032.png (166.0 KiB)
9 Answers
jacola avatar image
jacola answered ·

Update, fresh in 2021.
Victron finally took this serious, and i had a very pleasant support and cooperation by one of their core engineers. We did some remote sessions and i received new firmware which to my knowledge now has also been released.
I learned a lot over the course of the last months about Victron, and am happy with my installation now. There will be at least four more multiplus added to it.

So this case can be considered closed. It needed a firmware fix.

Johannes

1 comment
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

thomasw-1 avatar image thomasw-1 commented ·

Hi Johannes,
Please have a short look in GX (menu MultiPlus - device) what firmware version solved that issue.

Thanks
Thomas

0 Likes 0 ·
Daniël Boekel (Victron Energy Staff) avatar image
Daniël Boekel (Victron Energy Staff) answered ·

Hi @jacola

is it ok if we look into your system? if so, what is the VRM id or system name? (a link to the VRM page is also good)

10 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

jacola avatar image jacola commented ·

sorry i answered as "Answer" .. 41839

since today (portugese time) about 10:30 the inverters do "what they want" again, despite giving them setpoints. green is actual power as reported by the inverter, yellow is setpoint. note 10:42 all 3 phases show strange behavior .. at this time now things are back to ok.

we have cloudy weather today with substantial short term changes in production ..

0 Likes 0 ·
screenshot-2020-09-14-at-113758.png (360.3 KiB)
jacola avatar image jacola commented ·

Hi Daniël, did you come to any conclusions ? The problem happens intermittent, and not always on all phases. For me it almost looks as if the inverters go "on their own". Interesting is that the situation eventually fixes itself. If i should log any other values let me know please ..

0 Likes 0 ·
Daniël Boekel (Victron Energy Staff) avatar image Daniël Boekel (Victron Energy Staff) ♦♦ jacola commented ·

Hi @jacola

As you know, we're looking at it, I'll answer you as soon as I know more.

0 Likes 0 ·
jacola avatar image jacola Daniël Boekel (Victron Energy Staff) ♦♦ commented ·

Hi Daniel, here another log of a longer problem happening today.
The trigger of the situation is probably the following:
phase 1 inverts, phase 2 and 3 invert. a pump goes on (remember: nothing connected to load side, all is grid assist) drawing 5.5kw (again, on grid side) at 14:13
So the setpoints change, obviously.

From my viewpoints ESS at this timestarts to go in a havoc cycle, producing unrelated to my setpoints and fortuntaely, today did not violate grid feed rules (it produces together with production and that can drive too much triggering a protector, putting us off grid, that happened before).
It recovered around 14:38 for no obvious reason, but fortunately :)

here the graph

let me know if i can provide any other help to find this.

johannes

0 Likes 0 ·
screenshot-2020-09-28-at-153356.png (369.2 KiB)
jacola avatar image jacola Daniël Boekel (Victron Energy Staff) ♦♦ commented ·

Hi Daniel,

to avoid any doubt about how i interface to ESS mode 3, (which i do via Modbus/TCP), i added a completly independent monitoring via MQTT.
A simple software which just captures the (actively) reported setpoint and power power per phase and throws it to the reporting platform every 5 seconds. here are the subscribes:

client:subscribe("N/xxxxxxxxx/vebus/261/Hub4/+/AcPowerSetpoint",2)

client:subscribe("N/xxxxxxxxx/vebus/261/Ac/ActiveIn/+/P",2)

client:subscribe("N/xxxxxxxxx/vebus/261/Ac/ActiveIn/+/S",2)

The result shows that the ESS itself has those crazy ideas about not following the setpoint. usually for about 20 mins before getting "sane" again by itself. Green is the setpoing .. you can see hat ESS does out of it .. especially bad are the situations where the result is a fast discharge of the battery by producing a lot (and if that happens while we have production we blow the feed-in limits).

This really needs to be solved.

Johannes



0 Likes 0 ·
screenshot-2020-10-05-at-110749.png (393.2 KiB)
Daniël Boekel (Victron Energy Staff) avatar image Daniël Boekel (Victron Energy Staff) ♦♦ jacola commented ·

Hi @jacola


You mention this happens when the battery is fully charged, could it be that there is a DC overvoltage that causes the inverter to feed-in to prevent this overvoltage?

0 Likes 0 ·
jacola avatar image jacola Daniël Boekel (Victron Energy Staff) ♦♦ commented ·

@Daniël Boekel (Victron Energy Staff)

Hi Daniël, that was in one event, the above one was with an SOC of 18%. It happens quite often (usually one or multiple times per day), and i haven't found any correlation with anything until now. happens over all battery charge states, and also happens both with the batteries being charged and discharged (as you can see from the various posted graphics).
Happy to provide you with grafana access so you can soom to any area of the graphs if that would be helpful, but i would expect you can get the same data via VRM too ?
Is there anything i can do to help with the diagnosis ? I was about to write software to detect the situation so i then could try to command the ESS to do something (if it still listens in that phase).


Johannes


0 Likes 0 ·
Pedro Lima avatar image Pedro Lima commented ·

situation is really bad now, Can you please check what's the problem?

0 Likes 0 ·
jacola avatar image jacola Pedro Lima commented ·

Hi Pedro, Victron, i might have an idea. Here is a completely unbearable situation, which starts when the battery is getting full. The ESS software starts to oscilllate between massive "production" while i have full sun power and excessive energy, driving my feed-in way above the allowance .. the only thing i could do is really stop sending any setpoints and waiting. As usual, graph shows communicated setpoint and the results.

I am going to set up another independent monitoring using MQTT (currently i read back the data via modbus/tcp, and verify also that the grid connection shows those results so they are real and not fake).

Johannes

0 Likes 0 ·
screenshot-2020-10-01-at-234355.png (751.0 KiB)
jacola avatar image jacola jacola commented ·

So there are definitely 2 problems ..
the one reported where ESS in mode 3 just goes for its own decisions and does not follow the setpoints, and then the case when the battery is full - instead of leaving it alone, ESS goes to heavy discharging, trying to cut me off the grid with attempted overcurrent :(


Here is another great graphic of ESS not following the setpoints .. the green line is the setpoints


i am pretty desparate to get this solved.
The reported setpoints and other values are taken from MQTT from the venus, and directly sent to a victoriametrics/grafana instance. Setpoints are provided via modbus/tcp. so they do arrive in venus otherwise they would not be visible on this chart.

0 Likes 0 ·
screenshot-2020-10-09-at-170402.png (253.6 KiB)
jacola avatar image
jacola answered ·

Hi Daniel, absoluteley !

https://vrm.victronenergy.com/installation/41839/dashboard ..

i guess 41839 is the number ;)

Feel free to play, and i am more than happy to play along if you need local help, i should stop sending setpoints or the like .. linux/modbus/control specialist here ;)

If you want to see actual setpoints and the resulting data, i have that all on a victoriametrics/grafana instance, i can send you a login for that.

Johannes


2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Pedro Lima avatar image
Pedro Lima answered ·

Dear Daniel,


My customer Johannes is having some really annoying problems with this situation. Voltage from the grid is sometimes quite on the limit, and this behavior makes it even worse, triggering some equipments off due to a out-of-limits AC voltage.


Im glad Johannes runs an industrial electronics company, so he knows bugs like this happen sometimes. But if he was a regular customer, that pays a lot of money for this inverters, batteries (he has around 40kWh of Pylontech's), and this happens it would be a big mess (and the customer would be right to complain or even maybe go to court over this).


Johannes was planning to upgrade the system further, adding more inverters, batteries, panels, etc and converting to an off-grid system, to avoid the grid issues, but Im pretty sure if this software reliability problems are not sorted very soon, he will think twice about going off-grid using Victron.


Let me know if we can help in any way.


Best regards

2 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

jacola avatar image jacola commented ·

Hi Pedro the situation is getting really bad now.
As of today when the battery is 100% charged, the ESS starts to "oscillate" between inverting thousands of watts (into the grid which already has to accept the solar production) and charging, constantly. Even if i stop to send setpoints, this "out of the blue starting to invert" continues for a minute before the system finally goes to idle.
See below graph. Despite ESS mode 3, the system things i t can do what it wants and invert, causing grid feed-in problems .. the green line is the actual performance (below shaded line is inverting, above is charging). remote console shows 99%/100% charged battery.
I need the system also with a charged battery to balance phases, so i should be able to set one phase to inverting if necessary and others to charge .. but i would not expect the system to deliberately invert without command.

Johannes

0 Likes 0 ·
screenshot-2020-09-29-at-171632.png (679.5 KiB)
Pedro Lima avatar image Pedro Lima jacola commented ·

Can someone from Victron please reply?!?


We have no idea what the problem is, clearly its a software fault.


Can someone let us know if we can do something on our side?!?


This are expensive equipments, this lack of replies and lack of support should not happen!

0 Likes 0 ·
PeterM avatar image
PeterM answered ·

What is the ESS setting for Multiphase regulation ? Is it single phase or all 3 phases ? What is the external device ?

1 comment
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

jacola avatar image jacola commented ·

i am controlling via modbus/tcp. the ESS is set to mode 3 (external control), so i should be able to command every phase separately by setpoint. and it mostly (almost always) works well, just "sometimes" the ESS goes havoc and does whatever it wants, ignoring the setpoints given by the external control (which is a little linux appliance, a barix barionet, connected with an ET340 and controlling loads also).

To be 200% sure there is no problem with that device, i have set up yet another system, using MQTT, to just monitor the ESS. This other "Monitoring only" sytem receives both the setpoints as well as the actual data from ESS via MQTT, push, and records those to a victoriametrics cloud instance with grafana frontend. So i consider that 100% neutral, independent, just monitoring.
It shows the same data .. the ESS eventually ignores the setpoints and just does what it wants. those "havoc" phases can take 10-20 minutes, and in that time, unfortunately, ESS might deliberately go into 4k "production" on each phase, which adds to the solar production and can (and does) blow my line due to (negative!) overload. It clearly violates the feed-in regulation.

Next i am going to try to stop this, by setting the ESS to "charge only" when that happens, let's see if it accepts this.

But i have to say i am quite frustrated with this .. an ESS which decides by itself to feed-in and empty the battery, despite other commands, is useless.
As all three multiplus show the same behavior at the same time interval (usually all three start to ignore the setpoints) i have to assume it's the ESS and not the multiplus, but hard for me to diagnose this further. Sincerely hoping for Victron staff to get this diagnosed and fixed. I was about to upgrade the system substantially ...


0 Likes 0 ·
jacola avatar image
jacola answered ·

well, not really an answer, just more information to those interested or which run into the same problems.

The situation continues to be UNRESOLVED. Last week my installer was here to update the Multiplus Inverters as that was advised by Victron.
The problem remains.

To recapitulate: every now and then the multiplus inverters (i have to assume it is them) do not follow the setpoints (ESS Mode 3) any more but do go "havoc", and in worst case they go to full power inverting/production, causing excess grid feed-in and being cut off the network. Sometimes they just go to "0". In both cases the situation resolves itself within about 20 minutes, if we are not thrown off the grid of course. There are NO grid faults reported, nothin in NS log.
Here is a graph of the latest occurence, yesterday. At 16:43 all three phases seem to get to that point where they dont charge and dont produce, not following the setoint. At 16.53 they go "havoc". i did a multiplus subsystem reset via remote console at 16:58 which brought them back to 0 but phase 2 and three continue with havoc mode.
That data is collected by listening to the setpoint and actual power values published via MQTT by Venus, so that should be what the system should see.

Anyone else has similar problems ?
Is there any solution to listen on the VE.Bus to see what is going on there ?
For me the problem MUST URGENTLY be solved, i might be forced by the power provider to shut the inverters down otherwise as they produce excess grid feed in eventually, beyond my contracted power limit (when added with the solar production).


Any advice ? Anyone with similar problems ? If you think you might have this too, talk to me i can help you to log like this into a cloud server.

Johannes



screenshot-2020-10-16-at-170449.png (270.8 KiB)
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

thomasw-1 avatar image
thomasw-1 answered ·

@jacola,

Today I performed some tests on my 3-phase system in mode 3 and the externally definable setpoints.


In principle the user definable setpoints work fine on my side with 3 exceptions:

  • A setting higher than 1.600W per phase is simply ignored, although battery could be charged with 4-times higher current at that point of time and MultiPlus-II inverters should support up to 3.500W (70A) charging power per phase.
  • In certain currently non-reproducable cases the submitted setting gets simply ignored. I have to re-submit the setting up to 4-times until it gets effective.
    Interestingly the setting gets effective on 2 phases, where the remaining phase does anything else but does not align to the requested value.
  • Sometimes the defined power is aligned within a second only, but sometimes it takes up to 10 seconds to reach the defined limit.
3 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

jacola avatar image jacola commented ·

Hi Thomas,
since about 10 days, the problem is finally solved for me. Victron finally got serious on the problem, and they really have good engineers i have to say. Did remote sessions with the responsible engineer, he identified a reason, provided (just before christmas) a new firmware for the multiplus and the problem so far hasn't shown up again.
Which makes me quite happy. I have learned a lot on the way, and will add more victron equipment now to my setup (at least four multiplus 5000). So i definitely recommend to upgrade your Multplusses.
In my setup, i had both venus and cerbo used over the last months, setpoints sent via both modbus/tcp and mqtt, and that all works flawless (i sniffed on the VE.bus to ensure the setpoints really make it to the inverters).

Cheers

Johannes

0 Likes 0 ·
thomasw-1 avatar image thomasw-1 jacola commented ·

Great news, Johannes!
What firmware has solved your issue?
I have currently 477 running on my MultiPlus-II (2623)
The new release 478 does not refer to a fic related on mode 3.

1609604445194.png


Cheers,
Thomas

0 Likes 0 ·
1609604445194.png (5.5 KiB)
ondra avatar image ondra jacola commented ·

Hi Johanes, please, what version of FW is this, is it 478, as Thomas writes above, or some individual FW for your installation? I have a similar problem at one of the power plants. Thank you Ondra

0 Likes 0 ·
thomasw-1 avatar image
thomasw-1 answered ·

Hi Johannes,

@Ondra and I would highly appreciate, if you could invest 20 seconds in looking on your GX what firmware version solved your problem.

The community has the purpose to multiply solutions and not to keep it only for oneself.

Cheers,
Thomas

2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

gnagflow avatar image
gnagflow answered ·

Hi Thomas,

i didn't read all your post, but maybe it helps, maybe in your system the max. inverterpower (see manual 2.d) is not set to "1" = no limit - that it doesnt take the setpoint > 1600W

kind regards.

1616138028682.png


1616138028682.png (41.0 KiB)
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.