question

Check that you have an large enough power supply for the Pi. If you use a few USB connections or perhaps a hub you need at least 2.5A PSU.

Did you auto-reboot the pi when connection to vrm is lost or for any other reason?

The pi has no RTC, so the timestamp of your logs are then 1970, and your data transferred to the VRM has this time stamps (epoch data). You may add a gps receiver or a rtc as time source when the internet is unavailable and you need to reboot your device. Without getting time from a ntp server over the internet or another local source you will loose the logs.

The logs will be fine as long as the pi remains running and is not rebooted , it counts the time internally during the offline phases.

Well done, easy peasy...

So this is an answer of sorts:

If you have a connected USB GPS module, and most do seem to work. One of the features of GPS is that is has to know the time quite accurately to work, and it get the time from the GPS satellites.

And this data is available in the messages being transmitted to the Venus device serially over the USB. You can see the data as it arrives:

cat /run/serial-starter/gps/ttyACM0

The data format for each line is well documented if you search online.

So here is what I have done.

/data/rc.local is easy, add this line:

/data/gps.sh > /var/log/gps-date &

Now the bit that took time to write and debug. Don't ask me how it works I did it a couple of years ago and shell scripting is arcane.

The script gps.sh is:

#!/bin/bash
#add this line to /data/rc.local
#/data/gps.sh > /var/log/gps-date &
N=0;
FILE=/run/serial-starter/gps/ttyACM0
echo "Waiting for GPS data"
while [ ! -e $FILE ] && [ $N -lt 50 ];
do
    sleep 1;
    ((N++))
    echo -n "."
done;
sleep 5;

if [ $N -ge 17 ]; then
   echo " GPS data not available is the device plugged in"
   exit 0;
fi

echo
echo "Device attached waiting for data"

input=$FILE
N=0

# check if GPS has FIX
LINE1='^\$..GGA,[0-9\.]+,[0-9\.]+,[NS],[0-9\.]+,[EW],([0-2])'
# Catch lines with date and time info
LINE2='^\$GPRMC'

FIX=""

# extract date and time from GPS packet

function extract {
[[ $1 =~ ^\$GPRMC,([0-9]{4})([0-9]{2})\.[0-9]{2},.,[0-9\.]+,[NS],[0-9\.]+,[EW],[0-9\.]*,[0-9\.]*,([0-9]{2})([0-9]{2})([0-9]{2}), ]]
  echo "found this"
  COMMAND="20${BASH_REMATCH[5]}${BASH_REMATCH[4]}${BASH_REMATCH[3]}${BASH_REMATCH[1]}.${BASH_REMATCH[2]}"
  echo command is
  date -s $COMMAND
exit 0
}

while IFS= read -r line && [ $N -lt 2000 ] 
do
#  echo "$line"
  [[ $line =~ $LINE1 ]] && FIX=${BASH_REMATCH[1]} && echo "GPS device has a fix after $N lines"
  [[ $line =~ $LINE2 ]] && [[ $FIX -eq 1 ]] && extract $line
  ((N++))
done < "$input"
echo "Timed out no date/time information in 2000 lines from GPS receiver" 

The log output looks like this:

Waiting for GPS data


Device attached waiting for data
GPS device has a fix after 1 lines
found this
command is
Sun Jul 24 19:25:48 UTC 2022

The fix after n lines will take longer (than 1) on a system that has just been switched on.

Hi all,

Just an update on this.

The time/date has been fine with the installation of the RTC module.

What has not improved it the gaps in the logging. I have now confirmed, across multiple RPi 3B+ running the last few releases of VenusOS; that the logging will stop after ~2,900-3,200 records are written to the SD card.

Am I missing something? There is no storage issue with ~11GB available.

I did see this in a post relating to the Venus GX:

Data logging: 
When connected to the internet, all data is sent to the VRM Portal. When there is no internet connection available, the Venus GX will store the data internally, up to 48 hours. By inserting a micro SD-card or USB stick, more data can be stored. These files can then be uploaded to the VRM Portal, or offline converted with the VictronConnect app, for analysis.

The 48hrs lines up with what I'm seeing.

I do not see this issue on real Cerbos with a USB storage device plugged in. I've got customer installations that have had 3 months+ of data stored.

Ben.

I believe I've resolved this by running:

/opt/victronenergy/swupdate-scripts/resize2fs.sh

The location where the logs are written has gone from 403.5MB to 9.2GB.

Unfortunately, this has not resolved the issue. :(

The partitions prior to running the script:

and after:

but the logs are still topping out at ~3,000 records/2 days.

Can anyone please assist? @mvader (Victron Energy)

@benc_ipind the VRM temporary store is not held in / but in /data/db directory. You have loads of storage there, so that is not the issue.

Looking at one of your screen shots you are getting error 153, that should give you an idea of what is happening. Take a look at this part of the VenusOS manual.

It says

Error #153 Connection error: this could indicate a routing problem. For details, check the shown error message:

So you need to scroll down on the VenusOS screen and see what the error message shown is. That will give you some indication of the issue. It could be a DNS issue, routing issue, etc. The manual has two suggestions on what an Error #153 could be.

The logger has a concept of "Internal storage" to /data (flash normally) and it will only keep logs there for 2 days and "External storage" to a sd-card or pen-drive. Since on a RPI the sdcard is the "Internal Storage" logs will be limited. An easy way to make it work is just insert a pen-drive (not tested).