question

mr-fly avatar image
mr-fly asked

Powershell Script to join CSV and add Date to CSV Export from VictronConnect

I have written a Powershell Script which will join the CSV Export Files from Victron and add the Date to a column. I will provide this as is and cannot guarantee that it works for all and with all future exports. But maybe it helps...

Copy the code into eg. Merge_Files.ps1 and load eg. in Powershell ISE

Please edit the variables at the top of the file.

I will not do any kind of support for the script! Its community based!

I am not at all affiliated with Victron and dont get paid or whatever for this.

<#
Version: 1.0
Created on: 08.06.2020
Author: Thorsten Lambrecht
Notes: 
IMPORTANT: Names of the CSV have to be: YYYY-MM-DD_AnyName.csv
Date in the Name has to be date of the export, as this is the calculation base!
The output file will by default be saved in the Script Directory
Version:
1.0 Start Version
#>

$CSVFilesPath="c:\temp\VictronExports\*.csv"
$OutFilePath=$PSScriptRoot

#Get the List of all CSV Files
$CSVFiles=Get-ChildItem -path $CSVFilesPath
#Work on the CSV File by File
$AllValues=foreach ($CSV in $CSVFiles) {
    #Import the Export-Date from the Filename of the file
    $DatumStr=($CSV.Name -split "_")[0]
    #Convert the Date to a PS Datetime
    $Datum=$DatumStr -as [datetime]

    #Import the current CSV and skip first element, as it is usually Day "0" and we dont know, 
    #if day 0 is the complete value, unless values were exported at the end of the day in the dark
    #if you dont want to skip line "0" comment "-Skip 1" Line with a #
    #Values will be converted to appropriate types, as the general import would put them to NoteProperty.
    #Dont know if the errors are really int values, as I never had errors!
    $CurrentFileImport=Import-Csv  -Path $CSV.FullName -Delimiter "," | `
        select-object `
        -Skip 1 `
        @{Name="Date";Expression={$Datum.AddDays(-($_.'Days ago'))}},`
        @{Name="Date exported";Expression={$Datum}},`
        @{Name="Days ago";Expression={[int32]$_.'Days ago'}},`
        @{Name="Yield(Wh)";Expression={[int32]$_.'Yield(Wh)'}},`
        @{Name="Consumption(Wh)";Expression={[int32]$_.'Consumption(Wh)'}},`
        @{Name="Max. PV power(W)";Expression={[int32]$_.'Max. PV power(W)'}},`
        @{Name="Max. PV voltage(V)";Expression={[Single]$_.'Max. PV voltage(V)'}},`
        @{Name="Min. battery voltage(V)";Expression={[Single]$_.'Min. battery voltage(V)'}},`
        @{Name="Max. battery voltage(V)";Expression={[Single]$_.'Max. battery voltage(V)'}},`
        @{Name="Time in bulk(m)";Expression={[int32]$_.'Time in bulk(m)'}},`
        @{Name="Time in absorption(m)";Expression={[int32]$_.'Time in absorption(m)'}},`
        @{Name="Time in Single(m)";Expression={[int32]$_.'Time in Single(m)'}},`
        @{Name="Last error";Expression={[int32]$_.'Last error'}},`
        @{Name="2nd last error";Expression={[int32]$_.'2nd last error'}},`
        @{Name="3rd last error";Expression={[int32]$_.'3rd last error'}},`
        @{Name="4th last error";Expression={[int32]$_.'4th last error'}}
        
    $CurrentFileImport
}

#Now Group the values by Date, so that we see, which dates are overlapping
$Grouped=$AllValues | group-object -Property Date

#Now make a clean list, which will take only the Date-Line with the highest Yield, as this seems the full value
$CleanedList=foreach ($Group in $Grouped) {
    ($Group.Group | sort-object -Property 'Yield(Wh)' -Descending)[0]
}

#Export both lists to files. "AllValues.csv" is more as a reference to see, which lines have been doubled
#Output will be sorted Ascending, if you want Descending, add "-Descending" 
$AllValues | Sort-Object -Property Date | Export-Csv $OutFilePath\AllValues.csv -NoTypeInformation -Force
$CleanedList | Sort-Object -Property Date | Export-Csv $OutFilePath\CleanedList.csv -NoTypeInformation -Force


VictronConnect
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

2 Answers
mr-fly avatar image
mr-fly answered ·

I just added something else to the script.

Please remember, I put this together with most less effort and did not check on everything if it is the best way to do it. If you have any changes, which make sense, drop a line here.

There will be another export-file which will have all not avalilabe dates filled with a null value:

<#
Version: 1.1
Created on: 08.06.2020
Author: Thorsten Lambrecht
Notes: 
IMPORTANT: Names of the CSV have to be: YYYY-MM-DD_AnyName.csv
Date in the Name has to be date of the export, as this is the calculation base!
The output file will by default be saved in the Script Directory
Version:
1.0 Start Version
1.1 Added Fillup function to add days missing with 0 values
#>

#Please change the Path to the folder with the CSV-Files
$CSVFilesPath="C:\Temp\Org\*.csv"
$OutFilePath=$PSScriptRoot

#Get the List of all CSV Files
$CSVFiles=Get-ChildItem -path $CSVFilesPath
#Work on the CSV File by File
$AllValues=foreach ($CSV in $CSVFiles) {
    #Import the Export-Date from the Filename of the file
    $DatumStr=($CSV.Name -split "_")[0]
    #Convert the Date to a PS Datetime
    $Datum=$DatumStr -as [datetime]

    #Import the current CSV and skip first element, as it is usually Day "0" and we dont know, 
    #if day 0 is the complete value, unless values were exported at the end of the day in the dark
    #if you dont want to skip line "0" comment "-Skip 1" Line with a #
    #Values will be converted to appropriate types, as the general import would put them to NoteProperty.
    #Dont know if the errors are really int values, as I never had errors!
    $CurrentFileImport=Import-Csv  -Path $CSV.FullName -Delimiter "," | `
        select-object `
        -Skip 1 `
        @{Name="Date";Expression={$Datum.AddDays(-($_.'Days ago'))}},`
        @{Name="Date exported";Expression={$Datum}},`
        @{Name="Days ago";Expression={[int32]$_.'Days ago'}},`
        @{Name="Yield(Wh)";Expression={[int32]$_.'Yield(Wh)'}},`
        @{Name="Consumption(Wh)";Expression={[int32]$_.'Consumption(Wh)'}},`
        @{Name="Max. PV power(W)";Expression={[int32]$_.'Max. PV power(W)'}},`
        @{Name="Max. PV voltage(V)";Expression={[Single]$_.'Max. PV voltage(V)'}},`
        @{Name="Min. battery voltage(V)";Expression={[Single]$_.'Min. battery voltage(V)'}},`
        @{Name="Max. battery voltage(V)";Expression={[Single]$_.'Max. battery voltage(V)'}},`
        @{Name="Time in bulk(m)";Expression={[int32]$_.'Time in bulk(m)'}},`
        @{Name="Time in absorption(m)";Expression={[int32]$_.'Time in absorption(m)'}},`
        @{Name="Time in Single(m)";Expression={[int32]$_.'Time in Single(m)'}},`
        @{Name="Last error";Expression={[int32]$_.'Last error'}},`
        @{Name="2nd last error";Expression={[int32]$_.'2nd last error'}},`
        @{Name="3rd last error";Expression={[int32]$_.'3rd last error'}},`
        @{Name="4th last error";Expression={[int32]$_.'4th last error'}}
        
    $CurrentFileImport
}

#Now Group the values by Date, so that we see, which dates are overlapping
$Grouped=$AllValues | group-object -Property Date

#Now make a clean list, which will take only the Date-Line with the highest Yield, as this seems the full value
$CleanedList=foreach ($Group in $Grouped) {
    ($Group.Group | sort-object -Property 'Yield(Wh)' -Descending)[0]
}

#Export both lists to files. "AllValues.csv" is more as a reference to see, which lines have been doubled
#Output will be sorted Ascending, if you want Descending, add "-Descending" 
$AllValues | Sort-Object -Property Date | Export-Csv $OutFilePath\AllValues.csv -NoTypeInformation -Force
$CleanedList | Sort-Object -Property Date | Export-Csv $OutFilePath\CleanedList.csv -NoTypeInformation -Force

#Now sort the list for another file to be generated, take the cleaned one for it, as we only need the highest values
$SortedList = $CleanedList | sort-object -Property Date
#Detect the smallest date in all lists, this is our minimum date
$LowestDate=$SortedList[0].Date

#Template Hashtable for all values that will be added with 0 Values
$EmptyLine=@{
    "Date" = $([datetime]::MinValue)
    "Date exported" = $([datetime]::MinValue)
    "Days ago" = 0
    "Yield(Wh)" = 0
    "Consumption(Wh)" = 0
    "Max. PV power(W)" = 0
    "Max. PV voltage(V)" = 0
    "Min. battery voltage(V)" = 0
    "Max. battery voltage(V)" = 0
    "Time in bulk(m)" = 0
    "Time in absorption(m)" = 0
    "Time in Single(m)" = 0
    "Last error" = 0
    "2nd last error" = 0
    "3rd last error" = 0
    "4th last error" = 0
}

#Start with lowest date
$Current=$LowestDate
#Create ArrayList for the output
$FilledUp=new-object -TypeName System.Collections.ArrayList
#Run day by day up to yesterda
while ($Current -lt (Get-Date))
{
    #Check if there is a dataset in the list for the day, we currently work on
    $SetforDay=$SortedList | where-object {$_.Date -eq $Current}
    if ($SetforDay)
    { # we have found da dataset so we will take it
        $FilledUp.Add($SetforDay)
    }
    else
    { # no dataset for this day, we will add a zero one
        # clone the hashtable and convert it to pscustomobject, then change date value to date worked on
        $AddValue=[pscustomobject]($EmptyLine.Clone())
        $AddValue.Date = $Current
        # Add the values to the arraylist
        $FilledUp.Add($AddValue)
    }
    #Increment current worked on day by 1
    $Current=$Current.AddDays(1)
}

$FilledUp | Sort-Object -Property Date | Export-Csv $OutFilePath\FilledUp.csv -NoTypeInformation -Force


2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

mr-fly avatar image
mr-fly answered ·

As the new version of the Victron Connect App adds the date into the export. This script will no longer work. Maybe I will change it to:
1. Merge the Files

2. Eliminate double entries and take the higher one (the export-day is usually not finished if you export over the day and not at darkness)

3. Add "0" Values for all days missing, so an excel import looks nicer.

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Related Resources