@echo off

    setlocal enableextensions

    call :getNextFilename "filename*.txt" nextFilename
    echo %nextFilename% 
    echo test > "%nextFilename%"

    call :getNextFilename "%cd%\filename*.txt" nextFilename
    echo %nextFilename% 
    echo test > "%nextFilename%"

    endlocal
    exit /b

:getNextFilename whatToSearch returnVariable
    setlocal enableextensions enabledelayedexpansion
    for /f %%a in ("$\%~1"
    ) do for /f "tokens=1,* delims=*" %%b in ("%%~nxa"
    ) do ( set "left=%%b" & set "right=%%c" )
    set "max=0" 
    for %%a in ("%~1"
    ) do for /f "tokens=1 delims=%left%%right% " %%b in ("%%~nxa"
    ) do for /f "tokens=* delims=0 " %%c in ("0%%~b" 
    ) do if %%~c geq !max! set /a "max=%%c+1"
    endlocal & set "%~2=%~dp1%left%%max%%right%" & exit /b

This should find the next file in sequence independently of the existence of holes in the numeration of the files. A path can be included or omitted. The * will be used as the placeholder for the numeration. BUT this will not work if files or included paths have "problematic" characters.

@echo off

    setlocal enableextensions

    call :getNextFilename "filename*.txt" nextFilename
    echo %nextFilename% 
    echo test > "%nextFilename%"

    call :getNextFilename "%cd%\filename*.txt" nextFilename
    echo %nextFilename% 
    echo test > "%nextFilename%"

    endlocal
    exit /b

:getNextFilename whatToSearch returnVariable
    setlocal enableextensions enabledelayedexpansion
    for /f %%a in ("$\%~1"
    ) do for /f "tokens=1,* delims=*" %%b in ("%%~nxa"
    ) do ( set "left=%%b" & set "right=%%c" )
    set "max=0" 
    for %%a in ("%~1"
    ) do for /f "tokens=1 delims=%left%%right% " %%b in ("%%~nxa"
    ) do for /f "tokens=* delims=0 " %%c in ("0%%~b" 
    ) do if %%~c geq !max! set /a "max=%%c+1"
    endlocal & set "%~2=%~dp1%left%%max%%right%" & exit /b

This should find the next file in sequence independently of the existence of holes in the numeration of the files. A path can be included or omitted. The * will be used as the placeholder for the numeration. BUT this will not work if files or included paths have "problematic" characters.

If the date/time of creation of the file can be considered, then this version can be optimized as

:getNextFilename whatToSearch returnVariable
    setlocal enableextensions disabledelayedexpansion
    for /f %%a in ("$\%~1"
    ) do for /f "tokens=1,* delims=*?" %%b in ("%%~nxa"
    ) do ( set "left=%%b" & set "right=%%c" )
    set "max=0" 
    for /f "delims=" %%a in ('dir /tc /o-d /b "%~1" 2^>nul'
    ) do for /f "tokens=1 delims=%left%%right% " %%b in ("%%~nxa"
    ) do for /f "tokens=* delims=0 " %%c in ("0%%~b" 
    ) do set /a "max=%%c+1" & goto done
    :done
    endlocal & set "%~2=%~dp1%left%%max%%right%" & exit /b

that will take the latest created instance of the file set.