I'm writing a parser for HTTP Authorization header (see RFC2616#14.8 and RFC2617#1.2). Note that I explicitly don't care about the base64-encoded syntax used by HTTP Basic authentication. I'm only interested in the auth-param syntax used by Digest authentication (to be more specific, I'm implementing a custom Authorization header similar to this question on SO). Basically, it's just a list of key=value pairs separated by commas and value could be quoted or unquoted.
Here's my code, which seems to parse the examples from the RFC just fine:
package com.example.sample;
import java.util.HashMap;
import java.util.Map;
import java.util.NoSuchElementException;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
public class AuthorizationHeaderParser {
    private static final String SEPARATORS = "()<>@,;:\\\\\"/\\[\\]?={} \t";
    private static final Pattern TOKEN_PATTERN = Pattern
            .compile("[[\\p{ASCII}]&&[^" + SEPARATORS + "]&&[^\\p{Cntrl}]]+");
    private static final Pattern EQ_PATTERN = Pattern.compile("=");
    private static final Pattern TOKEN_QUOTED_PATTERN = Pattern
            .compile("\"([^\"]|\\\\\\p{ASCII})*\"");
    private static final Pattern COMMA_PATTERN = Pattern.compile(",");
    private static final Pattern LWS_PATTERN = Pattern
            .compile("(\r?\n)?[ \t]+");
    private static class Tokenizer {
        private String remaining;
        public Tokenizer(String input) {
            remaining = input;
        }
        private void skipSpaces() {
            Matcher m = LWS_PATTERN.matcher(remaining);
            if (!m.lookingAt()) {
                return;
            }
            String match = m.group();
            remaining = remaining.substring(match.length());
        }
        public String match(Pattern p) {
            skipSpaces();
            Matcher m = p.matcher(remaining);
            if (!m.lookingAt()) {
                return null;
            }
            String match = m.group();
            remaining = remaining.substring(match.length());
            return match;
        }
        public String mustMatch(Pattern p) {
            String match = match(p);
            if (match == null) {
                throw new NoSuchElementException();
            }
            return match;
        }
        public boolean hasMore() {
            skipSpaces();
            return remaining.length() > 0;
        }
    }
    public static Map<String, String> parse(String input) {
        Tokenizer t = new Tokenizer(input);
        Map<String, String> map = new HashMap<String, String>();
        String authScheme = t.match(TOKEN_PATTERN);
        map.put(":auth-scheme", authScheme);
        while (true) {
            while (t.match(COMMA_PATTERN) != null) {
                // Skip null list elements
            }
            if (!t.hasMore()) {
                break;
            }
            String key = t.mustMatch(TOKEN_PATTERN);
            t.mustMatch(EQ_PATTERN);
            String value = t.match(TOKEN_PATTERN);
            if (value == null) {
                value = t.mustMatch(TOKEN_QUOTED_PATTERN);
                // trim quotes
                value = value.substring(1, value.length() - 1);
            }
            map.put(key, value);
            if (t.hasMore()) {
                t.mustMatch(COMMA_PATTERN);
            }
        }
        return map;
    }
    public static void main(String args[]) {
        String test1 = "Digest\n"
                + "                 realm=\"[email protected]\",\n"
                + "                 qop=\"auth,auth-int\",\n"
                + "                 nonce=\"dcd98b7102dd2f0e8b11d0f600bfb0c093\",\n"
                + "                 opaque=\"5ccc069c403ebaf9f0171e9517f40e41\"";
        String test2 = "Digest username=\"Mufasa\",\n"
                + "                 realm=\"[email protected]\",\n"
                + "                 nonce=\"dcd98b7102dd2f0e8b11d0f600bfb0c093\",\n"
                + "                 uri=\"/dir/index.html\",\n"
                + "                 qop=auth,\n"
                + "                 nc=00000001,\n"
                + "                 cnonce=\"0a4f113b\",\n"
                + "                 response=\"6629fae49393a05397450978507c4ef1\",\n"
                + "                 opaque=\"5ccc069c403ebaf9f0171e9517f40e41\"";
        System.out.println(parse(test1));
        System.out.println(parse(test2));
    }
}
My questions:
- For something as simple as this, is my approach (using regex) good enough or should I write a "real" parser?
 - Is my translation from the RFC BNF to regex correct, or have I made any mistakes that fail on a valid header or pass an invalid header?
 - The regular expressions seem too complex, can they be simplified?
 - Any other suggestions?