CodeQL CLI may be slow to run when the codeql.zip is extracted at $HOME

[ghost]

In the Getting started with the CodeQL CLI documentation, it instructs the user to extract the downloaded zip archive, but doesn't advise where this should be:

2. Extract the zip archive

For Linux, Windows, and macOS users (version 10.14 "Mojave", and earlier) simply extract the zip archive.

[...]

3. Launch codeql

Once extracted, you can run CodeQL processes by running the codeql executable in a couple of ways:

• By executing <extraction-root>/codeql/codeql, where <extraction-root> is the folder where you extracted the CodeQL CLI package.

[...]

If the user chooses to extract in a directory tree which may contain a very large number of files and subdirectories, such as their home directory, then upon running any codeql command that resolves extensions, all of these subdirectories will be searched for files with certain filename patterns, which may take a very long time.

For example (output is from running kill -3 on the java process):

"pool-1-thread-1" #22 prio=5 os_prio=0 cpu=40821.78ms elapsed=40.94s tid=0x00007f3dec64a460 nid=0x1bd12 runnable  [0x00007f3dac3e3000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.fs.UnixNativeDispatcher.lstat0(java.base@17.0.7/Native Method)
	at sun.nio.fs.UnixNativeDispatcher.lstat(java.base@17.0.7/UnixNativeDispatcher.java:308)
	at sun.nio.fs.UnixFileAttributes.get(java.base@17.0.7/UnixFileAttributes.java:72)
	at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(java.base@17.0.7/UnixFileAttributeViews.java:52)
	at sun.nio.fs.UnixFileSystemProvider.readAttributes(java.base@17.0.7/UnixFileSystemProvider.java:148)
	at sun.nio.fs.LinuxFileSystemProvider.readAttributes(java.base@17.0.7/LinuxFileSystemProvider.java:99)
	at java.nio.file.Files.readAttributes(java.base@17.0.7/Files.java:1851)
	at java.nio.file.FileTreeWalker.getAttributes(java.base@17.0.7/FileTreeWalker.java:220)
	at java.nio.file.FileTreeWalker.visit(java.base@17.0.7/FileTreeWalker.java:277)
	at java.nio.file.FileTreeWalker.next(java.base@17.0.7/FileTreeWalker.java:374)
	at java.nio.file.Files.walkFileTree(java.base@17.0.7/Files.java:2845)
	at com.semmle.frontend.packs.DiskFileSystem.directChildren(DiskFileSystem.java:56)
	at com.semmle.frontend.packs.glob.PathMatchTreeWalker.pushStackEntry(PathMatchTreeWalker.java:345)
	at com.semmle.frontend.packs.glob.PathMatchTreeWalker.matchDirectory(PathMatchTreeWalker.java:205)
	at com.semmle.frontend.packs.glob.PathMatchTreeWalker.walk(PathMatchTreeWalker.java:121)
	at com.semmle.frontend.packs.glob.PathMatcher.walkFileTree(PathMatcher.java:270)
	at com.semmle.frontend.packs.ScanByManifests.scanTree(ScanByManifests.java:354)
	at com.semmle.frontend.packs.ScanByManifests.scanForPacksAndWorkspaces(ScanByManifests.java:154)
	at com.semmle.frontend.packs.QlPackResolver.scanForPacksAndWorkspaces(QlPackResolver.java:107)
	at com.semmle.frontend.packs.QlPackContainer.computePacksAndWorkspaces(QlPackContainer.java:119)
	at com.semmle.frontend.packs.QlPackContainer.lambda$new$0(QlPackContainer.java:38)
	at com.semmle.frontend.packs.QlPackContainer$$Lambda$145/0x0000000800d8dd70.apply(Unknown Source)
	at com.semmle.api.compilation.CachedOperation.get(CachedOperation.java:91)
	- locked <0x0000000591e21578> (a java.lang.Object)
	at com.semmle.frontend.packs.QlPackContainer.findPacksByName(QlPackContainer.java:69)
	at com.semmle.frontend.packs.QlPackSearchContext.findPacksByName(QlPackSearchContext.java:698)
	at com.semmle.frontend.packs.QlPackSearchContext.findPackByName(QlPackSearchContext.java:604)
	at com.semmle.frontend.packs.QlPackSearchContext.findPackByVersion(QlPackSearchContext.java:414)
	at com.semmle.frontend.packs.QlPackSearchContext.lambda$findAllPackDependencies$19(QlPackSearchContext.java:509)
	at com.semmle.frontend.packs.QlPackSearchContext$$Lambda$248/0x0000000800db9560.get(Unknown Source)
	at java.util.Optional.orElseGet(java.base@17.0.7/Optional.java:364)
	at com.semmle.frontend.packs.QlPackSearchContext.lambda$findAllPackDependencies$20(QlPackSearchContext.java:495)
	at com.semmle.frontend.packs.QlPackSearchContext$$Lambda$246/0x0000000800db90d0.apply(Unknown Source)
	at java.util.Optional.map(java.base@17.0.7/Optional.java:260)
	at com.semmle.frontend.packs.QlPackSearchContext.findAllPackDependencies(QlPackSearchContext.java:467)
	at com.semmle.cli2.resolve.ResolveExtensionsCommand.resolvePackDependencies(ResolveExtensionsCommand.java:267)
	at com.semmle.cli2.resolve.ResolveExtensionsCommand.executeJSON(ResolveExtensionsCommand.java:70)
	at com.semmle.cli2.resolve.ResolveExtensionsCommand.executeJSON(ResolveExtensionsCommand.java:43)
	at com.semmle.cli2.picocli.SimpleJsonSubcommand.executeInsistingOnJSONWithExistingMessages(SimpleJsonSubcommand.java:130)
	at com.semmle.cli2.picocli.SimpleJsonSubcommand$$Lambda$66/0x0000000800cf0a88.apply(Unknown Source)
	at com.semmle.cli2.picocli.SubcommandCommon.withCompilationMessages(SubcommandCommon.java:412)
	at com.semmle.cli2.picocli.SimpleJsonSubcommand.executeInsistingOnJSON(SimpleJsonSubcommand.java:119)
	at com.semmle.cli2.picocli.PlumbingRunner.call(PlumbingRunner.java:53)
	at com.semmle.cli2.picocli.SubcommandCommon.callPlumbingInProcess(SubcommandCommon.java:160)
	at com.semmle.cli2.test.CodeqlTestDir$ExtensionValuesCache.calculateExtensions(CodeqlTestDir.java:63)
	at com.semmle.cli2.test.CodeqlTestDir$ExtensionValuesCache.get(CodeqlTestDir.java:53)
	- locked <0x0000000591e21a18> (a java.util.concurrent.ConcurrentHashMap)
	at com.semmle.cli2.test.CodeqlTestDir.resolveExtensions(CodeqlTestDir.java:199)
	at com.semmle.cli2.test.CodeqlTestDir$$Lambda$118/0x0000000800d83990.run(Unknown Source)
	at java.util.concurrent.CompletableFuture$UniRun.tryFire(java.base@17.0.7/CompletableFuture.java:787)
	at java.util.concurrent.CompletableFuture.postComplete(java.base@17.0.7/CompletableFuture.java:510)
	at java.util.concurrent.CompletableFuture$AsyncRun.run(java.base@17.0.7/CompletableFuture.java:1810)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.7/ThreadPoolExecutor.java:1136)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.7/ThreadPoolExecutor.java:635)
	at java.lang.Thread.run(java.base@17.0.7/Thread.java:833)

The root cause seems to be that method com.semmle.cli2.resolve.ResolveExtensionsCommand.resolvePackDependencies calls com.semmle.frontend.packs.QlPackSearchContext.findAllPackDependencies with useLegacyResolution set to true, which ends up conducting a search of the parent of the codeql directory.

Please could you either change this behaviour, or make the documentation clearer about where the extraction root for codeql.zip should be to avoid this issue?

[MathiasVP]

Thanks for raising this issue. We will make sure to update the documentation to make users aware of this footgun 👍.