Anyway to import easy list into untangle?

[Guest]

Before importing I try to make an export of the current filters. I get a 2 byte file!!!
I think something in this app is broken.....

[Guest]

Tried. Same thing. Freezes upon hitting save.

[R. Shackleford]

I've be playing around with this for a couple days now. Seems the original scrip gave a JSON format that is different then the one I get if export a list from Untangle Ad Blocker. When Imported the results, they were also marked in Red indicating a "Delete" of the rules so when you click save, it process and you have a blank list.

I made some changes to the JSON output to match a test example I generated by manually adding one entry then exporting it.
My edits to the sting and order on lines 13 and 33 has resulted in a better JSON but one small problem persists. When it converts from the CVS to JSON, the values are getting put in "" example:"blocked":"true","flagged":"true". The two values that are true should not have the "" around them. This causes a JSON extension error when you try to save.

Anyone know how to convert the CVS to JSON with those values not in ""?

Here's my edited version. I changed the strings on lines 13 and 33 to match the JSON export of my user added test.

##
## Script Create by WebFooL for The Untangle Community
##
$easylistsource = "https://easylist.to/easylist/easylist.txt"
$Request = Invoke-WebRequest $easylistsource
$EasyList = $Request.Content
$filenamejson = "ADImport.json"
$filenamecsv = "ADImport.csv"
$easylistsourcecount=($EasyList | Measure-Object –Line).Lines
$hash = $null
$counter = 0
$hash = @'
blocked,flagged,string,javaClass,name,description,readOnly,id,category,enabled


'@


write-host "Will now work for a while do not panic!"
ForEach ($line in $($EasyList -split "`n"))
{
#Add Nice Progress bar.. 
Write-Progress -Activity "Processing Easylist" -CurrentOperation $line -PercentComplete (($counter / $easylistsourcecount) * 100)
    #Remove all Commented lines (They all start with !)
    if($line -clike '!*') {
    #Do Nothing
    } elseif($line -eq "[Adblock Plus 2.0]"){
    #Do Nothing
    }elseif($line -eq "" ){
    #Do Nothing
    }else {
        #Create Untangle JSON
        $hash += "true,true,$line,com.untangle.uvm.app.GenericRule,null,null,null,null,null,true`r`n"
        $counter++
    }    
}
#Tempstore as CSV
$hash | Set-Content -Path $filenamecsv


function Split-CsvFile {
    param (
        [string]$sourceCSV,
        [int]$size
    )


    $exportPath = $sourceCSV.Substring(0, $sourceCSV.LastIndexOf('.'))
    $count = (Import-Csv $sourceCSV).count
    $startrow = 0;
    $counter  = 1;


    while ($startrow -lt $count) {
        $filename = "$($exportPath)_$($counter)"
        Write-host "Filename:" $filename
        Import-CSV $sourceCSV | select-object -skip $startrow -first $size | Export-CSV "$filename.csv" -NoClobber -Encoding UTF8 -NoTypeInformation
        #Convert to Json
        import-csv "$filename.csv" | ConvertTo-Json -Compress | Set-Content -Path "$filename.json"
        if (Test-Path "$filename.csv") {
          Remove-Item "$filename.csv"
        }
        $startrow += $size
        $counter++
    }
}


Split-CsvFile -sourceCSV $filenamecsv -size 500 
#import-csv $filenamecsv | ConvertTo-Json -Compress | Set-Content -Path $filenamejson
#Count lines in the CSV
$numberoflines = (Import-Csv $filenamecsv | Measure-Object -Property string).Count
#Write friendly exit message
Write-Host "Done you now have a few $filenamejson with $numberoflines lines from $easylistsource"

[R. Shackleford]

Turned out my issues were being caused by MS Excel delimiter separating lines at the commas. I manually ingested Easylist into Excel without the delimiter, scrubbed the ! lines and used the CONCATENATE function to consolidate the few lines Excel still wants to separate. I edited down WebFool's scrip to just break out the CSV file I created into JSON files. Good news is, I don't really have to separate them and was successful at importing 50K lines (5,5583KB of data) with no issue.

[ccdmnk]

Does untangle even work with easylist formats? My tests with || doesn't seem to block anything. I will have to read up (or get help here) on which formats work with the web filter.

I wrote a script for adguards normal list which is less intrusive and has less bloat. This could easilly be rewritten for any other list if you know the untangle formatting.

Regex to format the lines to work with untangle rules (link to source in code, so you can modify it for whatever lists you find).
Regex match anything to exclude lines.

https://github.com/returaxel/UntangleAble-WebFilter/blob/main/untangleable-webfilter.ps1

It iterates easylist in 1-2 seconds on a decent machine.

[ccdmnk]

I totally missed that this topic was discussing AD BLOCKER app and not Web Filter. But alas it was too late for me to edit the previous reply. The link i sent earlier can upload the normal domain lists to the Web Filter app.

This script does what webfools' script does but a tad bit faster. Tested and can confirm it uploads the easylist OT without issues. Ta i trä!

https://github.com/returaxel/untangleable-webfilter/blob/main/untangleable-adblock.ps1

[MP715]

Do I want to import this list into the Ad Blocker or Web filter app or both?

[jcoffin]

Do not import into Web Filter. Frankly, there is no reason to run Ad Blocker if you have Web Filter.

[ccdmnk]

https://github.com/returaxel/untangleable-json

One can import domain lists to webfilter and the other can take the easy list and import in Adblocker. Tested and verified with 3 lists for webfilter and easylist for adblocker, noted in the code.

Doubt all of the domains in the filters are already in Web Filter app, although quite many of them seem to be already.

But since this is a manual or at the very least semi manual job and domain lists change it could very well screw you over if adding a bunch of lists. I wanted the AdGuard list in there as now i can ditch my AdGuard VM.

Any other reason not to import domain lists into webfilter @jcoffin?

Edit: might add, the default app list in web filter catches basically every ad on the pages i visit regularly.

[jcoffin]

Any other reason not to import domain lists into webfilter @jcoffin?

Webfilter uses a method similar to DNS lookup to determine if to block or not based on it's DB. Using blocking domain list is a glob operation. For a few it is harmless, but for 1000s of entries it creates latency and overtaxing of the CPU. Glob is not efficient.

[Lord Waffles]

I've be playing around with this for a couple days now. Seems the original scrip gave a JSON format that is different then the one I get if export a list from Untangle Ad Blocker. When Imported the results, they were also marked in Red indicating a "Delete" of the rules so when you click save, it process and you have a blank list.

I made some changes to the JSON output to match a test example I generated by manually adding one entry then exporting it.
My edits to the sting and order on lines 13 and 33 has resulted in a better JSON but one small problem persists. When it converts from the CVS to JSON, the values are getting put in "" example:"blocked":"true","flagged":"true". The two values that are true should not have the "" around them. This causes a JSON extension error when you try to save.

Anyone know how to convert the CVS to JSON with those values not in ""?

Here's my edited version. I changed the strings on lines 13 and 33 to match the JSON export of my user added test.

Just to clarify this, for anyone confused like I was, the problem is the JSON file being created from WebFool's script has all the TRUE and FALSE values wrapped in "". This is causing Untangle to crash when trying to save. A super simple fix is to simply run WebFools original script & then run a powershell script to replace all the "true" & "false" strings with just true & false (no "")

$original_file = 'c:\temp\ADImport.json'
$destination_file =  'c:\temp\ADImport.json'
(Get-Content $original_file) | Foreach-Object {
    $_ -replace '"true"', 'true' `
       -replace '"false"', 'false'
   
	 } | Set-Content $destination_file

Obviously change the file location for $original_file & $destination_file to your file/location. That's it! You should be able to upload all 60k lines to untangle without any issue.

Cheers!