Make a GET request to the following URL, you will get two fields name WorkflowInstanceID and WorkflowVersion along with other columns.
/_api/web/lists/GetByTitle('Your Library Name')/Fields
So you can get Workflow Instance Ids for all items in your library by following URL
/_api/web/lists/GetByTitle('Your Library Name')/Items?$select=WorkflowInstanceID
For a particular Item, let say ID of the Item is 11
/_api/web/lists/GetByTitle('Your Library Name')/Items(11)?$select=WorkflowInstanceID