I am trying to find the number of rows in a csv file that have above a certain value. The code I have goes something like
$T6=Import-Csv $file | Where-Object {$_."Value" -ge 0.6 } | Measure-Object
This works well for smaller files but for large csv files(1 GB or more) it will run forever. Is there any better way to parse csv files like this in powershell?
Import-Csv is the official cmdlet for this. One comment though, everything imported is a string, so you better cast the Value property to the correct type. For instance:
$T6 = Import-Csv $file | Where-Object { [float]$_.Value -ge 0.6 } | Measure-Object
You can try to get rid of Import-Csv:
$values = ([System.IO.File]::ReadAllText('c:\pst\New Microsoft Office Excel Worksheet.csv')).Split(";") | where {$_ -ne ""}
$items = New-Object "System.Collections.Generic.List[decimal]"
foreach($value in $values)
{
[decimal]$out = New-Object decimal
if ([System.Decimal]::TryParse($value, [ref] $out))
{
if ($out -ge 10){$items.Add($out)}
}
}
$items | Measure-Object
For speed when processing large files consider using a streamreader, Roman's answer here demonstrates usage.
