how to load a folder given just a ServerRelativeUrl using Microsoft.SharePoint.Client

In my application I have a url of a folder as retrieved by folder.ServerRelativeUrl() eg. /consulting/TeamSiteTemplate/Documents/

Now I want to get a list of folders inside this folder. I use this code:

using (ClientContext clientContext = new ClientContext(siteUrl))
{
   folder = clientContext.Web.GetFolderByServerRelativeUrl(folder.ServerRelativeUrl);
   clientContext.Load(folder);
   clientContext.Load(folder.Folders);
   clientContext.ExecuteQuery();

   foreach (Folder childFolder in folder.Folders)
   {
      // process folder
   }
}

This only seems to work when siteUrl contains the site eg. http://sharepoint/consulting/TeamSiteTemplate - but what if I don't know the site name of the given url? Or don't know if it has a subsite, etc.

So simply:

How can I get a site/subsite part from a relative url - or just the site if no subsite?

or: Is there another way to do this?