-1

I am using curl in a bash script to fetch the HTTP headers response:

curl -I https://github.com
HTTP/1.1 200 OK
server: GitHub.com
date: Sun, 07 Jun 2020 06:24:11 GMT
content-type: text/html; charset=utf-8
status: 200 OK
vary: X-PJAX, Accept-Encoding, Accept, X-Requested-With
etag: W/"97a0f898499532092ef829552032f2df"
cache-control: max-age=0, private, must-revalidate
strict-transport-security: max-age=31536000; includeSubdomains; preload
x-frame-options: deny
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
expect-ct: max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"

I want to get the above response in the json format.

I tried by adding the Content-Type header in the curl command:

curl -I -header "Content-Type: application/json" https://github.com

But I am not able to see the header response in the json format.

Please help.

1
  • Kindly improve your question, add code tag for your code as well. Commented Jun 7, 2020 at 6:35

1 Answer 1

1
curl -h | grep -e --header
 -H, --header <header/@file> Pass custom header(s) to server

So --header is to pass custom header(s) to the server. Not to have the response-header be returned as JSON. With the right JSON tools you'd have to do it yourself. I'd suggest you give a try.

Now it really depends on what you mean by "the json format". If all you want is the list of headers to be individual members in a JSON array, then this would do:

$ xidel -s --method=HEAD "https://github.com" -e 'array{$headers[.]}'
[
  "HTTP/1.1 200 OK",
  "Server: github.com",
  "Date: Thu, 20 Mar 2025 12:49:30 GMT",
  "Content-Type: text/html; charset=utf-8",
  "Vary: X-PJAX, X-PJAX-Container, Turbo-Visit, Turbo-Frame, Accept-Language,Accept-Encoding, Accept, X-Requested-With",
  "content-language: en-US",
  "ETag: W/\"0c9499e9e08f25ec542a64ebf8d3631b\"",
  "Cache-Control: max-age=0, private, must-revalidate",
  "Strict-Transport-Security: max-age=31536000; includeSubdomains; preload",
  "X-Frame-Options: deny",
  "X-Content-Type-Options: nosniff",
  "X-XSS-Protection: 0",
  "Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin",
  "Content-Security-Policy: default-src 'none'; [...] img-src 'self' data",
  "Accept-Ranges: bytes",
  "Set-Cookie: _gh_sess=[...]; Path=/; HttpOnly; Secure; SameSite=Lax",
  "Set-Cookie: _octo=GH1.1.880431740.1742474976; Path=/; Domain=github.com; Expires=Fri, 20 Mar 2026 12:49:36 GMT; Secure; SameSite=Lax",
  "Set-Cookie: logged_in=no; Path=/; Domain=github.com; Expires=Fri, 20 Mar 2026 12:49:36 GMT; HttpOnly; Secure; SameSite=Lax",
  "X-GitHub-Request-Id: EE8A:62E88:2E4D37:2F3CCC:67DC0EE0"
]

[.] is to remove empty lines (or items strictly speaking, because $headers holds a sequence). Without it the array would end like this:

  "X-GitHub-Request-Id: EE8A:62E88:2E4D37:2F3CCC:67DC0EE0"
  ""
]

If you still want to use curl, then you can pipe its output to xidel:

$ curl -sI "https://github.com" | xidel -se 'array{x:lines($raw)[.]}'

(x:lines($raw) is a shorthand for tokenize($raw,'\r\n?|\n') and turns $raw, the raw input, into a sequence where every new line is another item.)

If you want a JSON object instead of the array, then that's possible, but it requires a few extra steps:

$ xidel -s --method=HEAD "https://github.com" -e 'map:merge(for $item in $headers[.] let $kv:=tokenize($item,": ") return if (exists($kv[2])) then {$kv[1]:$kv[2]} else {"response":$kv[1]},{"duplicates":"combine"})'
$ xidel -s --method=HEAD "https://github.com" -e '
  map:merge(
    for $item in $headers[.]
    let $kv:=tokenize($item,": ")
    return
    if (exists($kv[2]))
    then {$kv[1]:$kv[2]}
    else {"response":$kv[1]},
    {"duplicates":"combine"}
  )
'
{
  "response": "HTTP/1.1 200 OK",
  "Server": "github.com",
  "Date": "Thu, 20 Mar 2025 12:49:30 GMT",
  "Content-Type": "text/html; charset=utf-8",
  "Vary": "X-PJAX, X-PJAX-Container, Turbo-Visit, Turbo-Frame, Accept-Language,Accept-Encoding, Accept, X-Requested-With",
  "content-language": "en-US",
  "ETag": "W/\"0c9499e9e08f25ec542a64ebf8d3631b\"",
  "Cache-Control": "max-age=0, private, must-revalidate",
  "Strict-Transport-Security": "max-age=31536000; includeSubdomains; preload",
  "X-Frame-Options": "deny",
  "X-Content-Type-Options": "nosniff",
  "X-XSS-Protection": "0",
  "Referrer-Policy": "origin-when-cross-origin, strict-origin-when-cross-origin",
  "Content-Security-Policy": "default-src 'none'; [...] img-src 'self' data",
  "Accept-Ranges": "bytes",
  "Set-Cookie": ["_gh_sess=[...]; [...] SameSite=Lax", "_octo=GH1.1.880431740.1742474976; [...] SameSite=Lax", "logged_in=no; [...] SameSite=Lax"],
  "X-GitHub-Request-Id": "EE8A:62E88:2E4D37:2F3CCC:67DC0EE0"
}
  • The if-statement is required, because the very first item, "HTTP/1.1 200 OK", doesn't have a key-name, so you'd have to add one.
  • In a JSON object duplicate keys are not allowed, so the 3 "Set-Cookie: [...]" items need to be reduced to just 1. The $options map for map:merge() does just that. For all duplicates a sequence is created. The "Set-Cookie" attribute now holds a sequence of the previously 3 separate items.
Sign up to request clarification or add additional context in comments.

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.