question

This sounds like a poor cable connection to the battery. Check for anything loose, poor crimps, proper wire gauge, etc. Are you chassis grounded or did you bring the negative to the battery as well?

To troubleshoot, compare the voltage drop at the inverter terminals to the voltage drop at the battery terminals, with a known load. Make sure your leads are on the busbar in the inverter, and then directly on the posts of the battery (not the clamps or lug) Run a sanity check on the numbers (is the vdrop across the wire what one would expect for that wire cross-section and length?)

All these written above have been done, cable connections have been checked while increasing battery capacity and based on that I decided to send the device to service. Let's see do they find anything from the device.

I have measured voltages from all separate batteries from their terminals to find out if the problem is there. I even made some new cables without good result. Groundings have been ok as well.

Guess it might be internal fault then! Hope it doesn't come back to you "no fault found", that's always frustrating!

The only other thing I could think is some kind of partial short on the AC output wiring or in one of your loads. BTW what are your test loads?

In my experience, Victron inverters have been sensitive to DC ripple, wire size, DC connection quality. Some other inverters are less picky, but I imagine it's at the cost of long term reliability (looser safety margins).