question

xzv avatar image
xzv asked

[CAN dropped messages] Multiplus II GX + Pylontech Force L2 = no data

I tried to build up two systems consisting of:

  • Multiplus II GX
  • Pylontech Force L2 (stack consists of 3x Battery module + 1x BMS)

I expected it to be "Plug & Play" but actually I'm now stuck for two days because I don't receive any detail data from the battery. The Multiplus recognizes a Pylontech battery but that's it. No SoC, no module level data.

as.png

s.png

However, when I unplug the CAN cable it's immediately reflected in the Remote Console. Plugging it back in brings the battery back.

It can be observed that it's very likely that it's a CAN or protocol error, as the status page suggests:

a.png

What I've done and tried so far:

  • Updated the GX and Multiplus to the latest firmware (v3.13).
  • Changed VE CAN to CAN-bus BMS (500 kbit/s).
  • Changing the termination resistor (even leaving it unplugged).
  • Following Victron's and Pylontech's manuals rigorously (here and here), paying special attention to pin mappings on both sides.
  • Made 3 different cables ("Type A"), each with or without a second CAN termination resistor.
  • Started the Multiplus first, or the Pylontech Force L2 first.
  • Waited for long times between boots.


The manufacturing date of the Pylontech Force L2 is Feb 2023, having installed firmware MW3_15_B505_1.27. Maybe it's a firmware problem on Pylontech side? I don't have access to a newer firmware.

My interpretation of the CAN port screenshot above with the partially lost TX data packets is that the Pylontech BMS is responding to most, but not all, of Victron's data requests. Therefore it could be a software problem on the Victron side or on the Pylontech side.

I saw similar posts in this forum dating a year back, but without a suggestion on how to solve this.

Any ideas, or someone who could fix this in the past?

Multiplus-IIcerbo gxPylontechVE.Can
as.png (23.2 KiB)
a.png (208.4 KiB)
s.png (23.8 KiB)
4 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

ejrossouw avatar image ejrossouw commented ·

@XZv For what it is worth try swapping the type A cable around. I had similar problem on a US3000C stack and that strangely solved it. Also, not sure if it applies, but Pylontech suggests not having the ground wire as it can interfere.

0 Likes 0 ·
xzv avatar image xzv ejrossouw commented ·
I already tried it without the GND wire connected. No luck so far :/
0 Likes 0 ·
ejrossouw avatar image ejrossouw xzv commented ·

Have you tried swapping the cable around using bms end at the Cerbo end?

0 Likes 0 ·
Show more comments
6 Answers
xzv avatar image
xzv answered ·

I tried the same cables and settings on my Cerbo GX and it works instantly. Any idea why it doesn't work with the Multiplus II GX on VE CAN?

1707655002600.png


1707655002600.png (163.4 KiB)
9 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

ejrossouw avatar image ejrossouw commented ·
@XZv Rather baffling and all that is left is to update the Cerbo firmware to the latest official 3.14 and then visit the Pylontech facebook support group if it still fails. What firmware version is the Cerbo GX on and are you using the VE.can or BMS.Can port?
0 Likes 0 ·
xzv avatar image xzv ejrossouw commented ·
Both have the latest 3.14 installed, so no difference there. The only difference I see is that the Cerbo GX has is dedicated BMS CAN port, where as I use the VE CAN port set to BMS CAN on the Multiplus II GX.
0 Likes 0 ·
ejrossouw avatar image ejrossouw xzv commented ·
I have a customer with a Force2 and Multiplus-II GX so this is most peculiar. There are actual differences between the dedicated "scaled down" BMS.can implementation of canbus and VE.can. On the latest MK II Cerbo however, there is only a VE.can port, which begs the question whether we will see more issues like this, not to mention the speed issues of BMS'es often operating at 500 kbits vs other devices at 250 kbits, which may limit the usefulness of a port intended for daisy chaining multiple devices. Would it be worth trying your luck with the 3.20.50 Beta firmware given it is easy to roll back?
0 Likes 0 ·
xzv avatar image xzv ejrossouw commented ·
Thanks for letting me know that you already saw it working in the field.


Good idea to try the beta. I'll let you know if it makes any difference. I can also try a downgrade, as the compatibility page suggests that v2.20 or later works with Force L2.
0 Likes 0 ·
xzv avatar image xzv ejrossouw commented ·

The upgrade to v3.20 beta didn't change the situation. Then I tried to downgrade the device to the version specified as compatible, v2.20. This downgrade bricked my GX. :(

Right now I have no idea how to bring it back to life.

0 Likes 0 ·
xzv avatar image xzv ejrossouw commented ·
@ejrossouw What firmware version does your customer use? Maybe that helps narrowing down what change in code (?) in Venus OS disabled the support of the Pylontech Force L2?
0 Likes 0 ·
ejrossouw avatar image ejrossouw xzv commented ·

@xzv I also sent an email to Pylontech support and await feedback. I noticed your other efforts and just wondered whether you tried both GX ports and checked the actual pins are ok?

0 Likes 0 ·
56d17f93-663d-47b2-801d-063e08f1a9e7.jpeg (322.5 KiB)
xzv avatar image xzv ejrossouw commented ·
Thanks, I'll try a downgrade to v3.10 later.

I checked all pins, I even exchanged the cables and Multiplus II GX to another one.

0 Likes 0 ·
xzv avatar image xzv ejrossouw commented ·

@ejrossouw I downgraded to v3.10 (the one your customer has installed). It doesn't change to the better, still no detail data. Very strange!

0 Likes 0 ·
Alex Pescaru avatar image
Alex Pescaru answered ·

Hi @XZv

Re bricked Cerbo:

See https://community.victronenergy.com/questions/204255/cerbo-gx-bricked-how-to-recover.html

Re CAN communication:

Cerbo doesn't "ask" for anything from battery. It just emits a specific presence ping on specific CAN IDs. In contrast, the battery is broadcasting its status over CAN and Cerbo just take the info from there.

Do you have superuser enabled on your Cerbo? Can you ssh log in into your Cerbo with the root user and password?

If yes, try to give the following command (without quotes): "candump can1" and report back here what you see there.

Of course, after you connected the battery cable on dedicated BMS CAN port, with proper terminator in the second BMS CAN port, because the BMS CAN port is "can1" interface in Cerbo hardware.

And also see if on the battery you have some specific dip switch configuration in order to enable CAN terminator there too.

Alex

2 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

xzv avatar image xzv commented ·

@Alex Pescaru

On the Multiplus II GX (port labelled "VE CAN"):

root@nanopi:/dev# candump can0
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  ...
root@nanopi:/dev# candump can1
SIOCGIFINDEX: No such device

Interesting enough is that there is no time delay when it prints that hundreds and hundreds of identical messages!

After attaching the CAN cable to the Cerbo GX (port labelled "BMS CAN"):

root@einstein:~# candump can1
  can1  305   [8]  00 00 00 00 00 00 00 00
  can1  307   [8]  12 34 56 78 56 49 43 00
  can1  305   [8]  00 00 00 00 00 00 00 00
  can1  307   [8]  12 34 56 78 56 49 43 00
  can1  359   [8]  00 00 00 00 03 50 4E 00
  can1  351   [8]  14 02 E8 03 E8 03 C2 01
  can1  355   [8]  38 00 64 00 00 00 00 00
  can1  356   [8]  35 13 FC FF C2 00 00 00
  can1  35C   [8]  C0 00 00 00 00 00 00 00
  can1  35A   [8]  AA AA AA 02 00 00 00 00
  can1  35E   [8]  50 59 4C 4F 4E 20 20 20
  can1  372   [8]  03 00 00 00 00 00 00 00
  can1  373   [8]  CE 0C CF 0C 24 01 25 01
  can1  374   [8]  30 31 30 31 00 00 00 00
  can1  375   [8]  30 31 30 32 00 00 00 00
  can1  376   [8]  30 31 30 32 00 00 00 00
  can1  377   [8]  30 31 30 31 00 00 00 00
  can1  379   [8]  DE 00 00 00 00 00 00 00
  can1  305   [8]  00 00 00 00 00 00 00 00
  can1  307   [8]  12 34 56 78 56 49 43 00
  can1  305   [8]  00 00 00 00 00 00 00 00
  can1  307   [8]  12 34 56 78 56 49 43 00
  can1  359   [8]  00 00 00 00 03 50 4E 00
  can1  351   [8]  14 02 E8 03 E8 03 C2 01
  can1  355   [8]  38 00 64 00 00 00 00 00
  can1  356   [8]  35 13 FD FF C2 00 00 00
  can1  35C   [8]  C0 00 00 00 00 00 00 00
  can1  35A   [8]  AA AA AA 02 00 00 00 00
  can1  35E   [8]  50 59 4C 4F 4E 20 20 20
  can1  372   [8]  03 00 00 00 00 00 00 00
  can1  373   [8]  CD 0C CF 0C 24 01 25 01
...


1 Like 1 ·
Alex Pescaru avatar image Alex Pescaru xzv commented ·

OK. @XZv

On the first case, on the VE CAN port, because the battery didn't see any inverter presence there, only broadcasts its own presence and tells "the world" that it's a Pylontech battery with 3 modules.

On the second case, on the BMS CAN port, because the inverter makes its presence felt there on the IDs 305 and 307, the battery starts to publish all the information it has available.

So, from the battery CAN communication point of view all is OK and all the info is there.

The problem, I'm afraid, it's on the Multiplus GX's side...

There the CAN profile should be set as CAN-BMS 500 kb/s.

And the battery won't start to send its data until it sees an inverter "ping" there and it seems that the nanopi doesn't send any presence pair/ping. Neither on 305 nor on 307.

Alex

1 Like 1 ·
xzv avatar image
xzv answered ·

Thanks for the analysis. Exactly because those two devices (Cerbo and Multiplus) show different behavior with the same firmware version (v3.14) I wanted to downgrade it to the version that is explicitly mentioned as being compatible to the Force L2.

Here's a comparison video of the Cerbo GX vs. the Multiplus II GX. First, I started candump on the Cerbo, then plugged in the CAN cable, waited for a few seconds before removing the cable. Then I did the same on the Multiplus II GX (candump, cable in, cable out). You can obviously see that the Multplus shows a strange behavior. No "presence" messages while disconnected, instead a load of messages from the battery once the cable is plugged in.

candump.mov

I read here that the serial bus could be a limitation in terms of throughput. But it's only one BMS, so that CAN bus is not congested...

I'm at a point where I doubt that I can generate more insights or debug it further.

Maybe @Matthias Lange - DE can comment on this? I'm pretty convinced that I found a bug or at least some unintended side effects. I'm happy to provide all information necessary to reproduce this, and as I mentioned before I'm not the only one affected.



candump.mov (51.2 MiB)
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

xzv avatar image
xzv answered ·

@Alex Pescaru

This is what dmesg has to say:

root@nanopi:~# dmesg | grep spi
[    2.373335] spi_master spi0: will run message pump with realtime priority
[    2.504228] mcp251xfd spi0.0 can0: MCP2518FD rev0.0 (+RX_INT -MAB_NO_WARN +CRC_REG +CRC_RX +CRC_TX +ECC -HD c:40.00MHz m:20.00MHz r:17.00MHz e:16.66MHz) successfully initialized.
[ 4130.521713] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4141.854706] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4152.854269] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4164.853296] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4174.855157] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4177.853663] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4199.853685] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4210.852941] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4220.850968] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4231.852828] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4241.853853] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4251.854103] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4262.853848] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4273.852063] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4284.853368] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4296.851815] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4306.854960] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4389.855231] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4394.858522] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4405.851135] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4415.854147] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4425.852498] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4437.851836] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
...

Any idea what goes wrong? Any options I can try on the CAN bus?


@Matthias Lange - DE This is not only my problem, as you can find similar cases in the forum like here. Can you please help?

2 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Matthias Lange - DE avatar image Matthias Lange - DE ♦ commented ·
I don't know anything about the CAN protocol, I forwarded it to Victron.
0 Likes 0 ·
xzv avatar image xzv Matthias Lange - DE ♦ commented ·

Thank you. Please let me know how I can reach someone from Victron to troubleshoot the problem further and fix it for everybody.

0 Likes 0 ·
xzv avatar image
xzv answered ·

Now I can share an update on my insights:

When I login via ssh and enter this:

root@nanopi:~# ip link set can0 down
root@nanopi:~# ip link set can0 up type can bitrate 501000

I get this:

present.png

In other words: If I set the baud rate manually from 500 to 501 kbit/s, the Multiplus II GX recognizes the battery.

This must be redone after each boot.

There must be an incorrect setting in Venus OS or a hardware timing problem.

How can I reach someone from Victron who can fix this?


present.png (80.6 KiB)
5 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Alex Pescaru avatar image Alex Pescaru commented ·

Hi @XZv

I believe you may have a dodgy CAN bus controller. Happens.

I am saying this because if it was a problem with the software it would appear in many more cases from the thousands equipment on the field.

Of course, in many cases engineers could implement in software workarounds for the hardware problems.

Anyway, glad that you are in the right direction and begin to see the light at the end of the tunnel. :-))

Alex


1 Like 1 ·
xzv avatar image xzv Alex Pescaru commented ·
Hi Alex,

I saw other cases in this forum already. I posted a link above. And I also exchanged the Multiplus and BMS with a replacement device, so it’s not a sibgle occurence.

I guess it could be that particular combination of hardware. But if it were in the majority of cases it would already be fixed :)

But I‘m glad I could identify at least a temporary workaround.

1 Like 1 ·
nickdb avatar image nickdb ♦♦ commented ·
Have you considered you have a BMS fault, maybe it was set incorrectly at the factory?

Have you contacted pylontech support?

There are thousands of pylon batteries in the field, and almost all battery BMS's run at 500kb, if the software was off by 1, many people would have problems.


0 Likes 0 ·
Guy Stewart (Victron Community Manager) avatar image Guy Stewart (Victron Community Manager) ♦♦ commented ·
Hi @XZv,

Thanks for persisting thru this and posting the results.

I am wondering what made you decide to try manually changing the baud rate to 501 kbps?

0 Likes 0 ·
xzv avatar image xzv Guy Stewart (Victron Community Manager) ♦♦ commented ·

Pure despair :)

I tried many other things but lead by intuition and interpretation of dropped TX messages I just gave this a try.

1 Like 1 ·
mvader (Victron Energy) avatar image
mvader (Victron Energy) answered ·

Hey @XZv , thank you for all reports and information. We’re looking into this and will come back to you.

6 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

xzv avatar image xzv commented ·
Thanks in advance!
0 Likes 0 ·
ejrossouw avatar image ejrossouw commented ·

There may be a link with this? VictronConnect on Android, MP 800VA using the MK3 dongle. Does not accepts the actual value.

0 Likes 0 ·
strange-behaviour.mp4 (6.9 MiB)
xzv avatar image xzv commented ·
Good morning @mvader (Victron Energy) ,

any updates on this? Could you reproduce it in your lab?

Thanks!

0 Likes 0 ·
ejrossouw avatar image ejrossouw xzv commented ·
Pylontech confirmed V1.6 is the latest firmware for the Force L2. Maybe get in touch with the reseller for Pylontech's local support information relevant to your location and also try the Pylontech Support on facebook.
0 Likes 0 ·
xzv avatar image xzv ejrossouw commented ·
I have doubt that it's on Pylontech's side. As stated above, two different Victron models either can deal with it, or not. As the software is the same, I conclude it must be a difference in the Victron hardware and its implication on the CAN communication.
0 Likes 0 ·
ejrossouw avatar image ejrossouw xzv commented ·
I agree ;)
0 Likes 0 ·