CMake Reusability and Package Management

My question is focused on CMake C++ projects and separating out code into multiple repositories for re-usability, and somewhat mimicking a naive package management system. For now I would like to ignore any possible package managers like Conan or vcpkg.

Suppose I have a C++ repository PackageA that uses CMake and supplies its own find package file FindPackageA.cmake so that it can be used in another CMake package. This file may do some checking to see if PackageA is already installed locally on the system (like its headers and lib files exist in some known location) and if not it will compile PackageA and install it locally (ideally this package would be versioned so it could find version 1, 2, ect. if requested).

I know have another package PackageB which wishes to consume PackageA. In theory, if both of these projects lived in a version control system like Github, I assume PackageB would have a CMakeLists.txt file that contains

...

# Download PackageA locally so that we get FindPackageA.cmake file
include(ExternalProject)
ExternalProject_Add(ProjectA
  GIT_REPOSITORY    MY_PACKAGE_A_GITHUB_REPO
  GIT_TAG           origin/release/1
)

# Run the FindPackageA.cmake file to get the headers and libs to link and use against.
find_package(PackageA)

...

My one problem with this potential solution is that it requires downloading PackageA every time to get its find package file. Is there a more maintainable way to separate out multiple CMake projects so that they can be easily resued here? Would it be better to have a single respository/package that just contains all the find package files so that they are separate from their respective packages?