One of the things I'm specifically wondering, is it ok/proper to have static helper functions in the .cpp file if they are not member functions? (should they be member functions and should they even be static?)

Yes, it is perfectly good. In fact, if something can be written using only the public interface, it should be written as a non-member.

Yes, functions used only in one CPP file should be static. In general, use an anonymous namespace for anything (including templates) used only in the one file.

Header

#include <experimental/filesystem>
#include <filesystem>

Both? Shouldn’t the regular one not exist if the experimental one does?

// With Visual Studio compiler, filesystem is still "experimental"
namespace fs = std::experimental::filesystem;

BTW, the experimental/filesystem has some differences from the standard. You can download the preview of the new version for free. I believe filesystem is conforming now, but have not verified that yet.

Yes! Very good. You did not make the common mistake of spilling the entire namespace into global space, and you abstracted where the filesystem is coming from. This can go inside the namespace, though. You are leaking this to users of the header file.

namespace imgui_ext {

Very good, putting your stuff in a namespace.

const struct file {
    std::string alias;
    fs::path path;
};

Didn’t that give you a compiler warning? Are you paying attention to any warnings and eliminating them at level 3? There is no variable being declared so the const doesn’t do anything here. Did you mean something else?

class file_browser_modal final {

There are no virtual functions declared, so final is unnecessary and misleading. (C++ is not Java)

static const int modal_flags;

I think this line is left over from an earlier edit. You are using a non-member variable of the same name in the CPP file instead. It is neither given any value nor defined anywhere.

const char* m_title;

What are you doing with a char*? I don’t know if it’s correct of advised yet, having only read to this point. But assuming you do indeed want a pointer to a primitive string info owned elsewhere, use a sgl::zstring (⧺SL.str.3)or std::string_view.

At this point, I wonder if the file struct can be nested inside file_browser_modal. No big deal if it’s easier to use this way, as it’s still inside imgui_ext.

rule of 5 ?

Now, you have a char* member, so I wonder how you copy, move, or delete the object? You did not define any special member functions.

file_browser_modal(const char* title);

The constructor ought to take a string_span or zstring as mentioned for the data member.

render

const bool render(const bool isVisible, std::string& outPath);

You are rare in over using const! The const on the return value doesn’t do anything. Did you mean to make this a const member instead?

The const on the isVisible parameter does not mean anything to the caller. You should leave it out here. You can add it in the definition where the body is; this does not change the signature.

I worry about outPath. Good that I can see it is meant to be an “out” parameter from the naming! But don’t make “out” parameters (⧺F.20.

I’ll return to this when I see what it does.

CPP file

regarding static

Putting static on a single function has been re-welcomed as just fine, in C++17. But you have several in a bunch, so use an anonymous namespace instead (which is still preferred).

static void get_files_in_path(const fs::path& path, std::vector<file>& files) {
    files.clear();

Don’t use “out” parameters to return information.

If returning files, the NRVO will be just as efficient (if not more) than what you wrote.

static std::vector<file> get_files_in_path(const fs::path& path) 
{
    std::vector<file> files;
      ⋮
    return files;
}

use auto (almost everywhere)

for (const fs::directory_entry& dirEntry : fs::directory_iterator(path)) {

It is very easy to not get the variable type exactly right, and end up copying each element when you meant to be referencing it, with no compiler warnings. So, it is especially important to use auto in a for loop.

static const int clamp_size_t_to_int(const size_t data) {
    static const int max_int = std::numeric_limits<int>::max();
    return static_cast<int>(data > max_int ? max_int : data);
}

Again, very good for thinking about narrowing issues.
It is said that “constexpr is the new static const”. So use it for max here.

BTW, just what will the promotion rules do if size_t is the same size as int (e.g. 4 bytes)? You should get a compiler warning here, so I suspect this is a lesson for you: as mentioned earlier, pay attention and eliminate warnings at level 3 (on MSVC).

static bool vector_file_items_getter(void* data, int idx, const char** out_text) {

The usual: don’t use an “out” parameter. I assume the void* is due to this being data kept by the GUI framework? Making this nicer is more advanced than you need right now. But use auto in the next line; you are naming the type twice.

I’m guessing the use of a pointer rather than the std::string or somesuch is another artefact of the GUI framework. It would be good if functions that need such “bad C++ due to API” are clearly labeled as such.

inline static constexpr int modal_flags =

inline or static? One or the other. constexpr implies static which is what you want here. (inline if used in a header, to get the linker to merge all the copies.)

file_browser_modal::file_browser_modal(const char* title) :

Good that you use an initialization list!
You are, however, using ye olde syntax. Use curlys, not parens, for initialization.

Since you only have the one constructor, “all” of them use the same values for most of the members. Put them in the class definition as direct initializers on the members, instead.

    ⋮
bool m_oldVisibility = false;
int m_selection = 0;
    ⋮

render

Same lessons explained already. Now how do you make a return value that can be a string or nothing? std::optional<std::string>.

This function is too long.
The body of “visibility state changes to visible” can go into a helper function. The body of the next if statement is huge; it can probably go into its own function which can then be split up into others.

Every time you have a comment header explaining what the next block of code does, you are “naming” that block of code and indicating it has a purpose distinct/different from other such blocks. A block of code with a cohesive single purpose and a name— what’s that normally called?

Closing remarks

Your code is a lot better than most of the code posted here. There are a lot of things about how-to that you already know. Your fluency in the language indicates that you are new to it, but you are learning from more up-to-date material compared with a lot of newcomers’ code, so good show there! Is there a book or online resource you recommend for this?