This depends on the value of history-limit that you have set in your .tmux.conf - the default is 2000; if you wish to capture more, you will need to explicitly set the number of lines.
To capture the entire scrollback, enter copy mode, select the entire scrollback, and yank it into the buffer, then paste it into your file.
How you accomplish this will depend on the mode-keys option you prefer, vi or emacs. man tmux has a helpful table describing the respective keys.
I have the following in my .tmux.conf to simplify this:
unbind [
bind Escape copy-mode
unbind p
bind p paste-buffer
bind-key -t vi-copy 'v' begin-selection
bind-key -t vi-copy 'y' copy-selection
The process for capturing the full scrollback is then:
PrefixEsc : to enter copy mode
v : to begin visual selection (assuming you are already at the bottom of the screen)
gg : to capture everything in the scrollback
y : to yank it into the buffer
Prefixc : open another tmux window
vim scrollback.txt
i : enter insert mode in vim
Prefixp : paste into file
There is also an answer here describing how to copy the buffer to a temporary file using xselcopy the buffer to a temporary file using xsel that might be useful.