Can't access SharePoint API from script but can from browser

Question

My site url looks like this

https://company.sharepoint.com/sites/ADV-DEV/assigDev/bassie/sitePages/home.aspx

I created a test list to try out the api and managed to get the list items using a SharePoint REST client addin and also managed to get that data by just going to this url in the browser

https://company.sharepoint.com/sites/ADV-DEV/assigDev/bassie/_api/lists/getByTitle('Test')/items

However, when trying to access the same from both PowerShell and C# I keep getting an error

403 Forbidden

$Cred = Get-Credential
$Url = "https://company.sharepoint.com/sites/ADV-DEV/assigDev/bassie/_api/lists/getByTitle('Test')/items"
Invoke-RestMethod -Method 'GET' -Uri $url -Credential $Cred

How is this possible? Is there some permission I need which is specifically for using the API from a script or program?

I tried getting my username with

https://company.sharepoint.com/sites/ADV-DEV/assigDev/bassie/_api/web/CurrentUser

Which shows loginname as

i:0#.f|membership|[email protected]

But I still get 403 when logging in with that

Answer 1

You can also try it as below:

Add-Type -Path 'C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.dll'
Add-Type -Path 'C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.Runtime.dll'

$siteUrl = "https://company.sharepoint.com/sites/ADV-DEV/assigDev/bassie"
$Username = "[email protected]"
$Password = "password"

$Context = New-Object Microsoft.SharePoint.Client.ClientContext($siteUrl)
$Context.Credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($Username, $(ConvertTo-SecureString -AsPlainText $Password -Force))
$Context.ExecuteQuery()    

$RequestUrl = "$($siteUrl)/_api/web/lists/getbytitle('Test')/items"
$AuthenticationCookie = $Context.Credentials.GetAuthenticationCookie($siteUrl, $true)

$WebSession = New-Object Microsoft.PowerShell.Commands.WebRequestSession
$WebSession.Credentials = $Context.Credentials
$WebSession.Cookies.SetCookies($siteUrl, $AuthenticationCookie)
$WebSession.Headers.Add("Accept", "application/json;odata=verbose")

$Result = Invoke-RestMethod -Method Get -WebSession $WebSession -Uri $RequestUrl

$Context.Dispose()

Write-Output $Result

Change your path of the dlls, site collection and list as per that in your environment.

If you dont have the files at that location, download and install the SharePoint Online Client Components SDK.

In case you need to perform POST queries, you will need to get the FormDigestValue. To do so, just add to previous $websession config:

$formDigest = Invoke-RestMethod -Method Post -WebSession $WebSession -Uri "$targetSiteUrl/_api/contextinfo" -errorAction Stop 
$WebSession.Headers.Add("X-RequestDigest", $formDigest.d.GetContextWebInformation.FormDigestValue)
$queryResult = Invoke-RestMethod -Method Post -WebSession $WebSession -Uri $restQuery-errorAction Stop

Download link - SharePoint Online Client Components SDK

Answer 2

RestSPO.ps1:

[Parameter(Mandatory=$True)]
[String]$UserName,

[Parameter(Mandatory=$False)]
[String]$Password
)

[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint.Client") | Out-Null
[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint.Client.Runtime") | Out-Null


if([string]::IsNullOrEmpty($Password)) {
   $SecurePassword = Read-Host -Prompt "Enter the password" -AsSecureString 
}
else {
   $SecurePassword = $Password | ConvertTo-SecureString -AsPlainText -Force
}


$credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($UserName, $SecurePassword)
$request = [System.Net.WebRequest]::Create($Url)
$request.Credentials = $credentials
$request.Headers.Add("X-FORMS_BASED_AUTH_ACCEPTED", "f")
$request.Accept = "application/json;odata=verbose"
$request.Method=$Method
$response = $request.GetResponse()
$requestStream = $response.GetResponseStream()
$readStream = New-Object System.IO.StreamReader $requestStream
$data=$readStream.ReadToEnd()
$results = $data | ConvertFrom-Json
$results.d.results 

and then invoke it using below:

 .\RestSPO.ps1 -Url "https://company.sharepoint.com/sites/ADV-DEV/assigDev/bassie/_api/lists/getByTitle('Test')/items" -UserName "[email protected]" -Password "password"

Answer 3

Another option I managed to get working was to first retrieve a security token from https://login.microsoftonline.com/extSTS.srf, and then an access token from https://COMPANY.sharepoint.com/_forms/default.aspx?wa=wsignin1.0:

$Global:Credential = Get-Credential -Message "cred" -UserName "[email protected]"

function Get-StringBetweenTwoStrings($firstString, $secondString, $inputString){
    #Regex pattern to compare two strings
    $pattern = "$firstString(.*?)$secondString"

    #Perform the opperation
    $result = [regex]::Match($inputString,$pattern).Groups[1].Value

    #Return result
    return $result
}
function Get-AuthCookies {
    $body = 
@"
<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope"
      xmlns:a="http://www.w3.org/2005/08/addressing"
      xmlns:u="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
  <s:Header>
    <a:Action s:mustUnderstand="1">http://schemas.xmlsoap.org/ws/2005/02/trust/RST/Issue</a:Action>
    <a:ReplyTo>
      <a:Address>http://www.w3.org/2005/08/addressing/anonymous</a:Address>
    </a:ReplyTo>
    <a:To s:mustUnderstand="1">https://login.microsoftonline.com/extSTS.srf</a:To>
    <o:Security s:mustUnderstand="1"
       xmlns:o="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
      <o:UsernameToken>
        <o:Username>$($Global:Credential.UserName)</o:Username>
        <o:Password>MyPassword</o:Password>
      </o:UsernameToken>
    </o:Security>
  </s:Header>
  <s:Body>
    <t:RequestSecurityToken xmlns:t="http://schemas.xmlsoap.org/ws/2005/02/trust">
      <wsp:AppliesTo xmlns:wsp="http://schemas.xmlsoap.org/ws/2004/09/policy">
        <a:EndpointReference>
          <a:Address>https://COMPANY.sharepoint.com/sites/ADV-DEV/assigDev/bassie/_api</a:Address>
        </a:EndpointReference>
      </wsp:AppliesTo>
      <t:KeyType>http://schemas.xmlsoap.org/ws/2005/05/identity/NoProofKey</t:KeyType>
      <t:RequestType>http://schemas.xmlsoap.org/ws/2005/02/trust/Issue</t:RequestType>
      <t:TokenType>urn:oasis:names:tc:SAML:1.0:assertion</t:TokenType>
    </t:RequestSecurityToken>
  </s:Body>
</s:Envelope>
"@

    $securityTokenUrl = "https://login.microsoftonline.com/extSTS.srf"
    $accessTokenUrl = "https://COMPANY.sharepoint.com/_forms/default.aspx?wa=wsignin1.0"

    $secTokenResponse = Invoke-RestMethod -Body $body -Uri $securityTokenUrl -Method POST

    $token = $secTokenResponse.Envelope.Body.RequestSecurityTokenResponse.RequestedSecurityToken.BinarySecurityToken.InnerText

    $accTokenResponse = Invoke-WebRequest -Body $token -Uri $accessTokenUrl -Method POST 

    $cookieHeaders = $accTokenResponse.Headers.'Set-Cookie'

    return $cookieHeaders
}

We then want to use these tokens to construct some cookies which we will send with the web request:

$cookie = Get-AuthCookies 
$rtFa = Get-StringBetweenTwoStrings -firstString "rtFa=" -secondString ";" -inputString $cookie
$FedAuth = Get-StringBetweenTwoStrings -firstString "FedAuth=" -secondString ";" -inputString $cookie

$url = "https://COMPANY.sharepoint.com/sites/ADV-DEV/assigDev/bassie/_api/lists/getByTitle('Test')/items"
[System.Uri]$Uri = $url
$ContentType = "application/json" 
$Method = 'GET' 

$Cookie1 = New-Object System.Net.Cookie
$Cookie1.Name = "rtFa" # Add the name of the cookie
$Cookie1.Value = $rtFa
$Cookie1.Domain = $uri.DnsSafeHost

$Cookie2 = New-Object System.Net.Cookie
$Cookie2.Name = "FedAuth" # Add the name of the cookie
$Cookie2.Value = $FedAuth
$Cookie2.Domain = $uri.DnsSafeHost

$WebSession = New-Object Microsoft.PowerShell.Commands.WebRequestSession
$WebSession.Cookies.Add($Cookie1)
$WebSession.Cookies.Add($Cookie2)

Finally, send the web request.

# Splat the parameters
$props = @{
    Uri         = $uri.AbsoluteUri
    Credential  = $Credential
    ContentType = $ContentType
    Method      = $Method
    WebSession  = $WebSession
}

$response = Invoke-RestMethod @props
$response.content.properties.Title

While this does work, it is pretty long-winded and unnecessarily complicated.

I ended up just using the Microsoft.SharePoint.Client library in C# to do what I needed - as all this needs is my credentials without the need to supply tokens..

I then realised I can use that same library in PowerShell with

$loadInfo1 = [System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint.Client")
$loadInfo2 = [System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint.Client.Runtime")