3

I need to get the same hash of an xml in any language.

I tried to get the xml's canonical form then get it's hash

But what I experienced was that the canonical is not a "fixed standard". It is implemented in different forms by all the libs and languages that I worked with... so, I never get the SAME hash.

So, my question is: is there a way to get a trustable hash of the same canonical XML?

Edit

I'm using this xml as example:

<doc xmlns="http://www.ietf.org" xmlns:w3c="http://www.w3.org" xml:base="something/else">
   <e1>
      <e2 xmlns="" xml:id="abc" xml:base="bar/">
         <e3 id="E3" xml:base="foo"/>
      </e2>
   </e1>
</doc>

To get the Canonical Form, I've used:

In c# (.net version 8):
string stringXml = "<doc xmlns=\"http://www.ietf.org\" xmlns:w3c=\"http://www.w3.org\" xml:base=\"something/else\">\n   <e1>\n      <e2 xmlns=\"\" xml:id=\"abc\" xml:base=\"bar/\">\n         <e3 id=\"E3\" xml:base=\"foo\"/>\n      </e2>\n   </e1>\n</doc>";

System.Security.Cryptography.Xml.XmlDsigC14NWithCommentsTransform c14n = new();

System.Xml.XmlDocument documentXml = new();
documentXml.LoadXml(stringXml);
c14n.LoadInput(documentXml);

Stream stream = (Stream)c14n.GetOutput(typeof(Stream));
string result = new StreamReader(stream).ReadToEnd();

using var hash = System.Security.Cryptography.SHA256.Create();
var byteArray = hash.ComputeHash(System.Text.Encoding.UTF8.GetBytes(result));
string sha256hex = Convert.ToHexString(byteArray);

Console.WriteLine(sha256hex);
  • The sha256hex result was:
4716238DE66819B69981AE1BD3943451D0EADEEA001583D27CDFDC4255484CB6
  • The canonicalized result was:
<doc xmlns="http://www.ietf.org" xmlns:w3c="http://www.w3.org" xml:base="something/else"><e1><e2 xmlns="" xml:base="bar/" xml:id="abc"><e3 id="E3" xml:base="foo"></e3></e2></e1></doc>
In java (version 21):
String stringXml= "<doc xmlns=\"http://www.ietf.org\" xmlns:w3c=\"http://www.w3.org\" xml:base=\"something/else\">\n   <e1>\n      <e2 xmlns=\"\" xml:id=\"abc\" xml:base=\"bar/\">\n         <e3 id=\"E3\" xml:base=\"foo\"/>\n      </e2>\n   </e1>\n</doc>";

org.apache.xml.security.Init.init();
org.apache.xml.security.c14n.Canonicalizer c14n = org.apache.xml.security.c14n.Canonicalizer.getInstance(org.apache.xml.security.c14n.Canonicalizer.ALGO_ID_C14N_WITH_COMMENTS);

java.io.ByteArrayOutputStream stream = new java.io.ByteArrayOutputStream();
c14n.canonicalize(stringXml.getBytes(), stream, false);
String result = stream.toString(java.nio.charset.StandardCharsets.UTF_8);
String sha256hex = org.apache.commons.codec.digest.DigestUtils.sha256Hex(result);
System.out.println(sha256hex);
  • The sha256hex result was:
dea874fbbe21f9e27e521cfddf61aa54bc1b0b18692e3105455eeca24beea1f6
  • The canonicalized result was:
<doc xmlns="http://www.ietf.org" xmlns:w3c="http://www.w3.org" xml:base="something/else">
   <e1>
      <e2 xmlns="" xml:base="bar/" xml:id="abc">
         <e3 id="E3" xml:base="foo"></e3>
      </e2>
   </e1>
</doc>
Improve this question
20
  • 2
    This is an interesting question. Canonical XML, on it surface, seems like it should eliminate inconsistencies, like white-space differences. Have you tried opening the two XML files in a diff tool to see what the differences are? Even differences in line endings could doom a trustable hash of the two files. Commented May 7, 2024 at 19:29
  • 1
    @gdonega, logical equivalence of two XML documents does not need to imply identical byte streams. Compare the canonicalized string that you feed into the hash algorithm and see if you can explain the differences in the byte stream. Be also on the lookout for things like a byte-order mark at the start of the byte stream. Commented May 8, 2024 at 7:20
  • 1
    @gdonega I would suggest trying a couple things: 1. print out the canonical forms and look for differences. 2. Strip out all unnecessary whitespace. Commented May 8, 2024 at 13:24
  • 2
    I just looked at the canonical spec and it does consider any whitespace within an element to be significant. That means all the line endings and well as indentation in your example doc. I see in the comments history that you tried stripping it out (sorry missed that before) but my experience says you should always do that. Commented May 8, 2024 at 13:46
  • 1
    I could have missed something but the only thing that I would expect to change when you canonicalize that document is that the <e3> element will be changed to: <e3 id="E3" xml:base="foo"></e3> See here. Commented May 8, 2024 at 13:51

2 Answers 2

Reset to default
4

Your problem is that in the C# code, prior to canonicalization, you parsed the XML document with a parser that strips whitespace, whereas in Java, you preserved the whitespace. In general whitespace in XML is significant, though of course in some particular document types it isn't. You need to decide whether to strip whitespace prior to canonicalization or not; you will get different signatures (and different canonicalizations) depending on whether you strip it or not.

With C#, to avoid stripping whitespace, set the PreserveWhitespace property on the XmlDocument.

If instead you want to add whitespace stripping to the Java code, the simplest way is to run a simple XSLT transformation:

<xsl:transform xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="3.0">
 <xsl:strip-space elements="*"/>
 <xsl:on-no-match="shallow-copy"/>
</xsl:transform>
Improve this answer
1
  • Thanks for your answer! I've tried your PreserveWhitespace solution, and I've experienced that even then the results were different. What did the trick (in my case) was to execute the regex "\W" (removing them) and then get the hash. It resolved my problems. Commented May 11, 2024 at 4:06
4

In both your C# and Java code, you are simply hashing the byte sequence contained in the string variable result. This will only yield the same hash value if both strings are byte for byte identical. Even a 1-bit difference in the two strings will produce completely different hash values. So you canonicalization procedure has to produce identical strings to give the same hash.

To diagnose the issue, save the string result to a file and compare with some sort of file comparison utility (Beyond Compare, WinDiff or even the CMD prompt's fc).

Improve this answer
2
  • I recently was writing a program that had to generate some XML documents that were binary identical to those produced by a commercial program. So I went through a lot of iterations of Beyond Compare. And I encountered a lot of WTFs by the commercial product with things like one part of the XML preferring <tag></tag> for empty tags, while other parts of the document preferring <tag />. Commented May 10, 2024 at 13:50
  • @PeterM That's exactly the problem that canonical XML aims to solve. Commented May 10, 2024 at 20:35

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.