The extension buffers progress messages from the CLI instead of showing them as they happen

[aschackmull]

Describe the bug
When the extension runs CodeQL CLI commands and those commands attempt to report progress through stderr then the messages are buffered and only shown after the CLI command has finished. This means that slow CLI commands cannot report progress when run through the VSCode extension.

Version
CodeQL extension version: 1.6.9
CodeQL CLI version: 2.10.2
Platform: darwin x64

To reproduce
Run any query. Then you'll see the CLI command succeeded. message before any [PROGRESS] messages from that CLI command. E.g. something like this:

Generating JSON log summary using CodeQL CLI: generate log-summary -v --log-to-stderr --format=predicates /Users/schack/Library/Application Support/Code/User/globalStorage/github.vscode-codeql/queries/quickquery.ql-d6L9VW0HE6rwNMfIW1ZHx/evaluator-log.jsonl /Users/schack/Library/Application Support/Code/User/globalStorage/github.vscode-codeql/queries/quickquery.ql-d6L9VW0HE6rwNMfIW1ZHx/evaluator-log.summary.jsonl...
CLI command succeeded.
[2022-08-10 11:07:09] [PROGRESS] generate log-summary> Beginning to generate summary for query log located at /Users/schack/Library/Application Support/Code/User/globalStorage/github.vscode-codeql/queries/quickquery.ql-d6L9VW0HE6rwNMfIW1ZHx/evaluator-log.jsonl
[2022-08-10 11:07:09] [PROGRESS] generate log-summary> Continuing to generate log summary: have currently processed about 99% of input file
[2022-08-10 11:07:09] [PROGRESS] generate log-summary> Finished generating log summary
[2022-08-10 11:07:09] [PROGRESS] generate log-summary> Beginning to generate summary for query log located at /Users/schack/Library/Application Support/Code/User/globalStorage/github.vscode-codeql/queries/quickquery.ql-d6L9VW0HE6rwNMfIW1ZHx/evaluator-log.jsonl
[2022-08-10 11:07:09] [PROGRESS] generate log-summary> Continuing to generate log summary: have currently processed about 99% of input file
[2022-08-10 11:07:09] [PROGRESS] generate log-summary> Finished generating log summary

Expected behavior
I expect the progress messages from the CLI command to be output while it is running and not be delayed until after it has finished.

Additional context
I believe the problem is here:

vscode-codeql/extensions/ql-vscode/src/cli.ts

Line 327 in c3a21b9

stderrBuffers.push(newData);

The stderr listener just adds the output to a buffer and only outputs it at the end in the finally block:

vscode-codeql/extensions/ql-vscode/src/cli.ts

Line 352 in c3a21b9

void this.logger.log(Buffer.concat(stderrBuffers).toString('utf8'));

Instead the stderr messages ought to go directly to the Extension Log output.

[aeisenberg]

One reason why this happens is that multiple CLI commands can be running at the same time. Outputting logs in this way minimizes the chance that multiple commands will have output interleaved. To be fair, though, I don't know how often this really happens.

[aschackmull]

Outputting logs in this way minimizes the chance that multiple commands will have output interleaved. To be fair, though, I don't know how often this really happens.

I don't think that's an issue. I'm definitely in favour of fixing this - even if it means some output interleaving in some cases (in most cases, I don't think it will).