General thoughts

You have managed to create a somewhat elegant API that balances between a state machine pattern and a regex engine. This is reusable for small and context-free use cases, but will get to haunt you if you need to tokenize more complex and context-bound grammars.

I can only add to VisualMelon's spot-on review:

• Tuples are fantastic constructs for internal data representation of an API, utility classes to avoid boiler-plate classes/structs. For the public connection points of any API however, they are more of a code smell. They somehow hurt readability. I feel a class name adds so much more to an input or result argument.
• There is room for improvement when dealing with escape characters and sequences. Currently only the double quote gets escaped. You could make a mini API for this.

Commandline API

Although this API is kept very simple, it already shows how you'd have to manage/corrupt your token design, just to be able to maintain simple regex patterns.

public enum CommandLineToken
{
    // .. other

    [Regex(@"[\=\:\,\s]")]
    ValueBegin,

    [QText(@"([a-z0-9\.\;\-]*)")]
    Value,
}

In my opinion, there should not be a distinction between ValueBegin and Value. They are both Value syntactically, only their semantics differ. I would never allow semantics to hurt my API design. This is a good example to show that regex only has benefits for the simpler grammars. Another proof to that point is that you required to make a custom pattern matcher QTextAttribute, because a regex would be too much pain to write (if even possible for balanced and escaped delimiters).

I like the API for its simplicity, and I see use cases for it. However, I'm afraid for most use cases, as more functionality is added over time, you'd end up with convoluted tokens and complex regexes to maintain. A next step is to ditch the regex engine and go for a full blown lexer.

Unnecessary switch-statements

switch statements are nice as a way of avoiding long chains of if (){} else if(){} .... else {} statements. Switching on a bool doesn't make much sense, as is much more unclear than using if statements. So replace this

switch (Escapables.Contains(c))
{
    case true:
        // Remove escape char.
        token.Length--;
        break;
}

for

if (Escapables.Contains(C))
{
    // Remove escape char.
    token.Length--;
}

and this

switch (i == offset)
{
    // Entering quoted text.
    case true:
        quote = !quote;
        continue; // Don't eat quotes.

    // End of quoted text.
    case false:
        return (true, token.ToString(), i - offset + 1);
}

for

if (i === offset)
{
    // Entering quoted text.
    quote = !quote;
    continue; // Don't eat quotes.
}
else 
{
    // End of quoted text.
    return (true, token.ToString(), i - offset + 1);
}

(self-answer)

I'll post another question when I made some more signifficant changes and for now I'll just summarize your feedback:

Suggestions by @VisualMelon

• ✔ - no public tuples (but one small extension) (you need to forgive me)
• ✔ - I must use the \G anchor more often; this simplfied the Regex matching
• ✔ - no more Making the compiler happy - removed weird switches
• ✔ - replaced mysterious return (false, token.ToString(), 0) with MatchResult<T>.Failure
• ✔ - Tokenize - a clean small while with a good looking switch
• ✔ - not generating tuples anymore; replaces with MatchResult<T>
• ✔ - State<TToken> is no longer restricted to Enum; instead, it now handles TToken via the new MatcherProviderAttribute that knows more about TToken and how to get IMatcher
• ✔ - MatchDelegate replaced with IMacher interface
• ✔/✖ - inline documentation - I'm trying ;-]

Suggestions by @dfhwze

• ✔ - both double and single quotes can be used; the first found is the one that must close a string
• ✔ - no more helper tokens like ValueBegin that weren't returned
• ✖ - context-bound grammars - maybe another time;
• ✖ - use a full blown lexer - maybe another time; for now this is fun

Suggestions by @JAD

• ✔ - no more switch flood

Conventions I might use some unusual conventions in my code and I think it's good to know them so that you're not surprised
- else if - this is worse than a goto
- is var x - I like this expression so I often use it to create inline variables
- ?: - I use this only for single expressions; who would want to debug a giant ternary; I prefer if/else with multiple conditions
- beware of var str = default(string) because I never define variables explicitly; this is not negotiable ;-P
- I use local functions to encapsulate small expressions
- I tend to (over)use System.Collections.Immutable because these classes have very convenient APIs
- I usually don't include parameter checking in proof-of-concept code

API

The Tokenizer is now only a small loop:

public interface ITokenizer<TToken> where TToken : Enum
{
    IEnumerable<Token<TToken>> Tokenize(string value);
}

public class Tokenizer<TToken> : ITokenizer<TToken> where TToken : Enum
{
    private readonly IImmutableDictionary<TToken, IImmutableList<State<TToken>>> _transitions;

    public Tokenizer(IImmutableList<State<TToken>> states)
    {
        _transitions = StateTransitionMapper.CreateTransitionMap(states);
    }

    public IEnumerable<Token<TToken>> Tokenize(string value)
    {
        var state = _transitions[default];
        var offset = 0;

        while (Any())
        {
            // Using a switch because it looks good here. 
            switch (state.Select(s => s.Match(value, offset)).FirstOrDefault(m => m.Success))
            {
                case null:
                    throw new ArgumentException($"Invalid character '{value[offset]}' at {offset}.");

                case MatchResult<TToken> match:
                    yield return new Token<TToken>(match.Token, match.Length, offset, match.TokenType);
                    offset += match.Length;
                    state = _transitions[match.TokenType];
                    break;
            }
        }

        // Let's hide this ugly expression behind this nice helper.
        bool Any() => offset < value.Length - 1;
    }
}

public static class StateTransitionMapper
{
    // Turns the adjacency-list of states into a dictionary for faster lookup.
    public static IImmutableDictionary<TToken, IImmutableList<State<TToken>>> CreateTransitionMap<TToken>(IImmutableList<State<TToken>> states) where TToken : Enum
    {
        return states.Aggregate(ImmutableDictionary<TToken, IImmutableList<State<TToken>>>.Empty, (mappings, state) =>
        {
            var nextStates =
                from n in state.Next
                join s in states on n equals s.Token
                select s;

            return mappings.Add(state.Token, nextStates.ToImmutableList());
        });
    }
}

Supporting types

All other supporting types implementing the changes listed in the summary above.

public class MatchResult<TToken>
{
    public MatchResult(string token, int length, TToken tokenType)
    {
        Success = true;
        Token = token;
        Length = length;
        TokenType = tokenType;
    }

    public static MatchResult<TToken> Failure(TToken tokenType) => new MatchResult<TToken>(string.Empty, 0, tokenType) { Success = false };

    public bool Success { get; private set; }

    public string Token { get; }

    public int Length { get; }

    public TToken TokenType { get; }
}

public interface IMatcher
{
    MatchResult<TToken> Match<TToken>(string value, int offset, TToken tokenType);
}

public abstract class MatcherAttribute : Attribute, IMatcher
{
    public abstract MatchResult<TToken> Match<TToken>(string value, int offset, TToken tokenType);
}

// Can recognize regexable patterns.
// The pattern requires one group that is the token to return. 
public class RegexAttribute : MatcherAttribute
{
    private readonly Regex _regex;

    public RegexAttribute([RegexPattern] string prefixPattern)
    {
        _regex = new Regex($@"\G{prefixPattern}");
    }

    public override MatchResult<TToken> Match<TToken>(string value, int offset, TToken tokenType)
    {
        return
            _regex.Match(value, offset) is var match && match.Success
                ? new MatchResult<TToken>(match.Groups[1].Value, match.Length, tokenType)
                : MatchResult<TToken>.Failure(tokenType);
    }
}

// Can recognize constant patterns.
public class ConstAttribute : MatcherAttribute
{
    private readonly string _pattern;

    public ConstAttribute(string pattern) => _pattern = pattern;

    public override MatchResult<TToken> Match<TToken>(string value, int offset, TToken tokenType)
    {
        return
            // All characters have to be matched.
            MatchLength() == _pattern.Length
                ? new MatchResult<TToken>(_pattern, _pattern.Length, tokenType)
                : MatchResult<TToken>.Failure(tokenType);

        int MatchLength() => _pattern.TakeWhile((t, i) => value[offset + i].Equals(t)).Count();
    }
}

// Assists regex in tokenizing quoted strings because regex has no memory of what it has seen.
// Requires two patterns:
// - one for the separator because it has to know where the value begins
// - the other for an unquoted value if it's not already quoted
public class QTextAttribute : MatcherAttribute
{
    public static readonly IImmutableSet<char> Escapables = new[] { '\\', '"', '\'' }.ToImmutableHashSet();

    private readonly Regex _prefixRegex;
    private readonly Regex _unquotedValuePattern;

    public QTextAttribute([RegexPattern] string separatorPattern, [RegexPattern] string unquotedValuePattern)
    {
        _prefixRegex = new Regex($@"\G{separatorPattern}");
        _unquotedValuePattern = new Regex($@"\G{unquotedValuePattern}");
    }

    public override MatchResult<TToken> Match<TToken>(string value, int offset, TToken tokenType)
    {
        if (_prefixRegex.Match(value, offset) is var prefixMatch && prefixMatch.Success)
        {
            if (MatchQuoted(value, offset + prefixMatch.Length, tokenType) is var matchQuoted && matchQuoted.Success)
            {
                return matchQuoted;
            }
            else
            {
                if (_unquotedValuePattern.Match(value, offset + prefixMatch.Length) is var valueMatch && valueMatch.Groups[1].Success)
                {
                    return new MatchResult<TToken>(valueMatch.Groups[1].Value, prefixMatch.Length + valueMatch.Length, tokenType);
                }
            }
        }

        return MatchResult<TToken>.Failure(tokenType);
    }

    // "foo \"bar\" baz"
    // ^ start         ^ end
    private static MatchResult<TToken> MatchQuoted<TToken>(string value, int offset, TToken tokenType)
    {
        var token = new StringBuilder();
        var escapeSequence = false;
        var quote = '\0'; // Opening/closing quote.

        foreach (var (c, i) in value.SkipFastOrDefault(offset).SelectIndexed())
        {
            if (i == 0)
            {
                if (@"'""".Contains(c))
                {
                    quote = c;
                }
                else
                {
                    // It doesn't start with a quote. This is unacceptable. Either an empty value or an unquoted one.
                    return MatchResult<TToken>.Failure(tokenType);
                }
            }
            else
            {
                if (c == '\\' && !escapeSequence)
                {
                    escapeSequence = true;
                }
                else
                {
                    if (escapeSequence)
                    {
                        if (Escapables.Contains(c))
                        {
                            // Remove escape char. We don't need them in the result.
                            token.Length--;
                        }

                        escapeSequence = false;
                    }
                    else
                    {
                        if (c == quote)
                        {
                            // +2 because there were two quotes.
                            return new MatchResult<TToken>(token.ToString(), i + 2, tokenType);
                        }
                    }
                }

                token.Append(c);
            }
        }

        return MatchResult<TToken>.Failure(tokenType);
    }
}

public static class StringExtensions
{
    // Doesn't enumerate the string from the beginning for skipping.
    public static IEnumerable<char> SkipFastOrDefault(this string source, int offset)
    {
        // Who uses for-loop these days? Let's hide it here so nobody can see this monster.
        for (var i = offset; i < source.Length; i++)
        {
            yield return source[i];
        }
    }

    // Doesn't enumerate a collection from the beginning if it implements `IList<T>`.
    // Falls back to the default `Skip`.
    public static IEnumerable<T> SkipFastOrDefault<T>(this IEnumerable<T> source, int offset)
    {
        // Even more for-loops to hide.
        switch (source)
        {
            case IList<T> list:
                for (var i = offset; i < list.Count; i++)
                {
                    yield return list[i];
                }

                break;

            default:
                foreach (var item in source.Skip(offset))
                {
                    yield return item;
                }

                break;
        }
    }
}

public static class EnumerableExtensions
{
    // This is so common that it deserves its own extension.
    public static IEnumerable<(T Item, int Index)> SelectIndexed<T>(this IEnumerable<T> source)
    {
        return source.Select((c, i) => (c, i));
    }
}

public abstract class MatcherProviderAttribute : Attribute
{
    public abstract IMatcher GetMatcher<TToken>(TToken token);
}

public class EnumMatcherProviderAttribute : MatcherProviderAttribute
{
    public override IMatcher GetMatcher<TToken>(TToken token)
    {
        if (!typeof(TToken).IsEnum) throw new ArgumentException($"Token must by of Enum type.");

        return
            typeof(TToken)
                .GetField(token.ToString())
                .GetCustomAttribute<MatcherAttribute>();
    }
}

public class State<TToken> where TToken : Enum
{
    private readonly IMatcher _matcher;

    public State(TToken token, params TToken[] next)
    {
        Token = token;
        Next = next;
        _matcher =
            typeof(TToken)
                .GetCustomAttribute<MatcherProviderAttribute>()
                .GetMatcher(token);
    }

    public TToken Token { get; }

    public IEnumerable<TToken> Next { get; }

    public MatchResult<TToken> Match(string value, int offset) => _matcher.Match(value, offset, Token);

    public override string ToString() => $"{Token} --> [{string.Join(", ", Next)}]";
}

public class Token<TToken>
{
    public Token(string token, int length, int index, TToken type)
    {
        Text = token;
        Length = length;
        Index = index;
        Type = type;
    }

    public int Index { get; }

    public int Length { get; }

    public string Text { get; }

    public TToken Type { get; }

    public override string ToString() => $"{Index}: {Text} ({Type})";
}

Tests & Examples

This is how I use it with a simplfied commad-line syntax:

using static CommandLineToken;

public class CommandLineTokenizerTest
{
    private static readonly ITokenizer<CommandLineToken> Tokenizer = new CommandLineTokenizer();

    [Theory]
    [InlineData(
        "command -argument value -argument",
        "command  argument value argument")]
    [InlineData(
        "command -argument value value",
        "command  argument value value")]
    [InlineData(
        "command -argument:value,value",
        "command  argument value value")]
    [InlineData(
        "command -argument=value",
        "command  argument value")]
    [InlineData(
        "command -argument:value,value",
        "command  argument value value")]
    [InlineData(
        @"command -argument=""foo--bar"",value -argument value",
        @"command  argument   foo--bar   value  argument value")]
    [InlineData(
        @"command -argument=""foo--\""bar"",value -argument value",
        @"command  argument   foo-- ""bar   value  argument value")]
    public void Can_tokenize_command_lines(string uri, string expected)
    {
        var tokens = Tokenizer.Tokenize(uri).ToList();
        var actual = string.Join("", tokens.Select(t => t.Text));
        Assert.Equal(expected.Replace(" ", string.Empty), actual);
    }
}

[EnumMatcherProvider]
public enum CommandLineToken
{
    Start = 0,

    [Regex(@"\s*(\?|[a-z0-9][a-z0-9\-_]*)")]
    Command,

    [Regex(@"\s*[\-\.\/]([a-z0-9][a-z\-_]*)")]
    Argument,

    [QText(@"([\=\:\,]|\,?\s*)", @"([a-z0-9\.\;\-]+)")]
    Value,
}

public class CommandLineTokenizer : Tokenizer<CommandLineToken>
{
    /*

     command [-argument][=value][,value]

     command --------------------------- CommandLine
            \                           /
             -argument ------   ------ /    
                      \      / \      /
                       =value   ,value

    */
    private static readonly State<CommandLineToken>[] States =
    {
        new State<CommandLineToken>(default, Command),
        new State<CommandLineToken>(Command, Argument),
        new State<CommandLineToken>(Argument, Argument, Value),
        new State<CommandLineToken>(Value, Argument, Value),
    };

    public CommandLineTokenizer() : base(States.ToImmutableList()) { }
}

A couple of tiny tidbits:

1. You could easily make Token immutable (removing the property setters) by passing type and index into the constructor as such:

   public Token((bool Success, string Token, int Length) match, TToken type, int index)
   {
       (bool success, string token, int length) = match;
       this.Length = success ? length : 0;
       this.Text = success ? token : string.Empty;
       this.Type = type;
       this.Index = index;
   }
   

then you just have to adjust Consume in the State class like so:

public Token<TToken> Consume(string value, int offset)
{
    return new Token<TToken>(_match(value, offset), Token, offset);
}

2. Token and State are, in my opinion, screaming to have their own interfaces:

    public interface IState<TToken> where TToken : Enum
    {
        bool IsToken { get; }

        TToken Token { get; }

        IEnumerable<TToken> Next { get; }

        IToken<TToken> Consume(string value, int offset);
    }

    public interface IToken<TToken> where TToken : Enum
    {
        int Length { get; }

        string Text { get; }
    }

(adjust accordingly in the bunch of places they're used)