Why can't I focus on certain elements using react refs?

I have a weird case where I have a list of refs of form elements including buttons, different kinds of inputs, etc. I have then created a summary of errors which when clicked on automatically focuses thus also scrolls to the element in need of attention. This works perfectly for input[type="text"] and for textareas, but input[type="file"] and buttons (type button) does not work without any errors. However, and this is the weird part, if I click on an element above and then tab down manually, bringing the focus on the element. The focus function then starts working again.

I simply get all the refs by calling an exposed function using imperativeHandle. The refs are created like this: const fileInput = createRef<HTMLInputElement>(); and assigned to the input element. The code is very simple, is this a known issue?

Thanks in advance.

EDIT: Tried replicating the error, and got the same problem with this simple code:

import React, { createRef } from 'react';
import logo from './logo.svg';
import './App.css';

function App() {
  const textRef = createRef<HTMLInputElement>();
  const textAreaRef = createRef<HTMLTextAreaElement>();
  const fileRef = createRef<HTMLInputElement>();
  const buttonRef = createRef<HTMLButtonElement>();
  return (
    <div className="App">
      <div>
        <input type="text" ref={textRef} />
      </div>
      <div>
        <textarea ref={textAreaRef} ></textarea>
      </div>
      <div>
        <input type="file" ref={fileRef}/>
      </div>
      <div>
        <button ref={buttonRef}>Button</button>
      </div>
      Links:
      <div>
        <button onClick={() => {
          textRef.current?.focus();
        }}>Text</button>
      </div>
      <div>
        <button onClick={() => {
          textAreaRef.current?.focus();
        }}>TextArea</button>
      </div>
      <div>
        <button onClick={() => {
          fileRef.current?.focus();
        }}>File</button>
      </div>
      <div>
        <button onClick={() => {
          buttonRef.current?.focus();
        }}>Button</button>
      </div>
    </div>
  );
}

export default App;

There is the same issue when using scrollTo.

EDIT 2: Tried adding preventDefault to the onclick functions but it did not have any effect. Navigating with the keyboard and pressing on the items using space or enter seems to work flawlessly, but not when clicking with the cursor.