Document library with only owner access using PnP PowerShell

Using PnP PowerShell, I would like to create an extra document library in SharePoint Online, where only the owner group has access. With the script below I am able to remove all groups I want to remove, except "Hub Visitors".

How can I get to the point where only the owner group has access to the document library?

$InternalDocumentsLibraryName = "Internal Docs"
New-PnPList -Title InternalDocumentsLibraryName -Template DocumentLibrary

$MemberGroupName = Get-PnPGroup -AssociatedMemberGroup
$VisitorGroupName = Get-PnPGroup -AssociatedVisitorGroup


$InternalDocuments = Get-PnPList -Identity $InternalDocumentsLibraryName
 
If($InternalDocuments)
{
       
    Set-PnPList -Identity $InternalDocumentsLibraryName -BreakRoleInheritance -CopyRoleAssignments
    Set-PnPListPermission -Identity $InternalDocumentsLibraryName -Group $MemberGroupName -RemoveRole 'Edit'
    Set-PnPListPermission -Identity $InternalDocumentsLibraryName -Group $VisitorGroupName -RemoveRole 'Read'
}
Else
{   
    Write-Host -f Yellow "Could not find '$InternalDocumentsLibraryName'"
}