390

I've been trying to learn Go on my own, but I've been stumped on trying read from and write to ordinary files.

I can get as far as inFile, _ := os.Open(INFILE, 0, 0), but actually getting the content of the file doesn't make sense, because the read function takes a []byte as a parameter.

func (file *File) Read(b []byte) (n int, err Error)

11 Answers 11

597

Let's make a Go 1-compatible list of all the ways to read and write files in Go.

Because file API has changed recently and most other answers don't work with Go 1. They also miss bufio which is important IMHO.

In the following examples I copy a file by reading from it and writing to the destination file.

Start with the basics

package main

import (
    "io"
    "os"
)

func main() {
    // open input file
    fi, err := os.Open("input.txt")
    if err != nil {
        panic(err)
    }
    // close fi on exit and check for its returned error
    defer func() {
        if err := fi.Close(); err != nil {
            panic(err)
        }
    }()

    // open output file
    fo, err := os.Create("output.txt")
    if err != nil {
        panic(err)
    }
    // close fo on exit and check for its returned error
    defer func() {
        if err := fo.Close(); err != nil {
            panic(err)
        }
    }()

    // make a buffer to keep chunks that are read
    buf := make([]byte, 1024)
    for {
        // read a chunk
        n, err := fi.Read(buf)
        if err != nil && err != io.EOF {
            panic(err)
        }
        if n == 0 {
            break
        }

        // write a chunk
        if _, err := fo.Write(buf[:n]); err != nil {
            panic(err)
        }
    }
}

Here I used os.Open and os.Create which are convenient wrappers around os.OpenFile. We usually don't need to call OpenFile directly.

Notice treating EOF. Read tries to fill buf on each call, and returns io.EOF as error if it reaches end of file in doing so. In this case buf will still hold data. Consequent calls to Read returns zero as the number of bytes read and same io.EOF as error. Any other error will lead to a panic.

Using bufio

package main

import (
    "bufio"
    "io"
    "os"
)

func main() {
    // open input file
    fi, err := os.Open("input.txt")
    if err != nil {
        panic(err)
    }
    // close fi on exit and check for its returned error
    defer func() {
        if err := fi.Close(); err != nil {
            panic(err)
        }
    }()
    // make a read buffer
    r := bufio.NewReader(fi)

    // open output file
    fo, err := os.Create("output.txt")
    if err != nil {
        panic(err)
    }
    // close fo on exit and check for its returned error
    defer func() {
        if err := fo.Close(); err != nil {
            panic(err)
        }
    }()
    // make a write buffer
    w := bufio.NewWriter(fo)

    // make a buffer to keep chunks that are read
    buf := make([]byte, 1024)
    for {
        // read a chunk
        n, err := r.Read(buf)
        if err != nil && err != io.EOF {
            panic(err)
        }
        if n == 0 {
            break
        }

        // write a chunk
        if _, err := w.Write(buf[:n]); err != nil {
            panic(err)
        }
    }

    if err = w.Flush(); err != nil {
        panic(err)
    }
}

bufio is just acting as a buffer here, because we don't have much to do with data. In most other situations (specially with text files) bufio is very useful by giving us a nice API for reading and writing easily and flexibly, while it handles buffering behind the scenes.


Note: The following code is for older Go versions (Go 1.15 and before). Things have changed (ioutil is deprecated since Go 1.16). For the new way, take a look at this answer.

Using ioutil

package main

import (
    "io/ioutil"
)

func main() {
    // read the whole file at once
    b, err := ioutil.ReadFile("input.txt")
    if err != nil {
        panic(err)
    }

    // write the whole body at once
    err = ioutil.WriteFile("output.txt", b, 0644)
    if err != nil {
        panic(err)
    }
}

Easy as pie! But use it only if you're sure you're not dealing with big files.

Sign up to request clarification or add additional context in comments.

20 Comments

For anyone who stumbles upon this question, it was originally asked in 2009 before these libraries were introduced, so please, use this answer as your guide!
According to golang.org/pkg/os/#File.Write, when Write hasn't written all bytes, it returns an error. So the extra check in the first example (panic("error in writing")) isn't necessary.
Note that these examples aren't checking the error return from fo.Close(). From the Linux man pages close(2): Not checking the return value of close() is a common but nevertheless serious programming error. It is quite possible that errors on a previous write(2) operation are first reported at the final close(). Not checking the return value when closing the file may lead to silent loss of data. This can especially be observed with NFS and with disk quota.
So, what's a "big" file? 1KB? 1MB? 1GB? Or does "big" depend on the machine's hardware?
@425nesp It reads whole the file into the memory, so it depends on the amount of available memory in the running machine.
|
67

New Way

Starting with Go 1.16, use os.ReadFile to load the file into memory, and use os.WriteFile to write to a file from memory (ioutil.ReadFile now calls os.ReadFile and is deprecated).

Be careful with the os.ReadFile because it reads the whole file into memory.

package main

import "os"
import "log"

func main() {
    b, err := os.ReadFile("input.txt")
    if err != nil {
        log.Fatal(err)
    }

    // `b` contains everything your file has.
    // This writes it to the Standard Out.
    os.Stdout.Write(b)

    // You can also write it to a file as a whole.
    err = os.WriteFile("destination.txt", b, 0644)
    if err != nil {
        log.Fatal(err)
    }
}

2 Comments

What if I have a 20GB file with 16GB of ram? 🙂
One option is to use io.Copy and stream.
58

This is a good version (but do note that ioutil is deprecated since Go 1.16):

package main

import (
  "io/ioutil"; 
  )


func main() {
  contents,_ := ioutil.ReadFile("plikTekstowy.txt")
  println(string(contents))
  ioutil.WriteFile("filename", contents, 0644)
}

4 Comments

This stores the whole file in memory. Since the file can be large, that may not be always what you want to do.
Also, 0x777 is bogus. In any case, it should be more like 0644 or 0755 (octal, not hex).
@cnst changed it to 0644 from 0x777
47

Using io.Copy

package main

import (
    "io"
    "log"
    "os"
)

func main () {
    // open files r and w
    r, err := os.Open("input.txt")
    if err != nil {
        panic(err)
    }
    defer r.Close()

    w, err := os.Create("output.txt")
    if err != nil {
        panic(err)
    }
    defer w.Close()

    // do the actual work
    n, err := io.Copy(w, r)
    if err != nil {
        panic(err)
    }
    log.Printf("Copied %v bytes\n", n)
}

If you don't feel like reinventing the wheel, the io.Copy and io.CopyN may serve you well. If you check the source of the io.Copy function, it is nothing but one of the Mostafa's solutions (the 'basic' one, actually) packaged in the Go library. They are using a significantly larger buffer than he is, though.

6 Comments

one thing worth to mention - to be sure that the content of the file was written to disk, you need to use w.Sync() after the io.Copy(w, r)
Also, if you write to already existing file, io.Copy() will only write the data you feed it with, so if existing file had more content, it won't be removed, which may result in corrupted file.
@Invidian It all depends on how you open the destination file. If you do w, err := os.Create("output.txt"), what you describe does not happen, because "Create creates or truncates the named file. If the file already exists, it is truncated." golang.org/pkg/os/#Create.
This should be the correct answer as it doesn't re-invent the wheel while not having to read the entire file at once before reading it.
@ShayTsadok Isn't closing the file will sync the data?
|
23

Note. ioutil is deprecated since Go 1.16.

With newer Go versions, reading/writing to/from a file is easy. To read from a file:

package main

import (
    "fmt"
    "io/ioutil"
)

func main() {
    data, err := ioutil.ReadFile("text.txt")
    if err != nil {
        return
    }
    fmt.Println(string(data))
}

To write to a file:

package main

import "os"

func main() {
    file, err := os.Create("text.txt")
    if err != nil {
        return
    }
    defer file.Close()

    file.WriteString("test\nhello")
}

This will overwrite the content of a file (create a new file if it was not there).

Comments

11

[]byte is a slice (similar to a substring) of all or part of a byte array. Think of the slice as a value structure with a hidden pointer field for the system to locate and access all or part of an array (the slice), plus fields for the length and capacity of the slice, which you can access using the len() and cap() functions.

Here's a working starter kit for you, which reads and prints a binary file; you will need to change the inName literal value to refer to a small file on your system.

package main
import (
    "fmt";
    "os";
)
func main()
{
    inName := "file-rw.bin";
    inPerm :=  0666;
    inFile, inErr := os.Open(inName, os.O_RDONLY, inPerm);
    if inErr == nil {
        inBufLen := 16;
        inBuf := make([]byte, inBufLen);
        n, inErr := inFile.Read(inBuf);
        for inErr == nil {
            fmt.Println(n, inBuf[0:n]);
            n, inErr = inFile.Read(inBuf);
        }
    }
    inErr = inFile.Close();
}

4 Comments

The Go convention is to check for error first, and let the normal code reside outside the if block
@Jurily: If the file is open when the error occurs, how do you close it?
But why is a [256]byte not accepted and the clearly silly and verbose (but apparently not wrong) inBuf:=make([]byte, 256) accepted?
I think it should be os.OpenFile and not os.Open?
9

Note. Since Go 1.16 ReadFile is in the os package.

Try this:

package main

import (
  "io"; 
  )
  

func main() {
  contents,_ := io.ReadFile("filename");
  println(string(contents));
  io.WriteFile("filename", contents, 0644);
}

5 Comments

This will work if you want to read the whole file at once. If the file's really big or you only want to read part of it, it might not be what you're looking for.
You should really check the error code, and not ignore it like that!!
This has been moved into the ioutil package now. So it would be ioutil.ReadFile()
I fixed so it says 0644
It seems it was moved again to the os package.
5

you can use the fmt package also:

package main

import "fmt"

func main(){
    file, err := os.Create("demo.txt")
    if err != nil {
        panic(err)
    }
    defer file.Close()
    
    fmt.Fprint(file, name)
}

Comments

1

Just looking at the documentation it seems you should just declare a buffer of type []byte and pass it to read which will then read up to that many characters and return the number of characters actually read (and an error).

The docs say

Read reads up to len(b) bytes from the File. It returns the number of bytes read and an Error, if any. EOF is signaled by a zero count with err set to EOF.

Does that not work?

EDIT: Also, I think you should perhaps use the Reader/Writer interfaces declared in the bufio package instead of using os package.

1 Comment

You have my vote because you actually acknowledge what real people see when they read the documentation, instead of parrotting what those accustomed to Go are REMINDED OF (not reading REMINDED OF) when they read the documentation of the function they are familiar with already.
1

The Read method takes a byte parameter because that is the buffer it will read into. It's a common idiom in some circles and makes some sense when you think about it.

This way you can determine how many bytes will be read by the reader and inspect the return to see how many bytes actually were read and handle any errors appropriately.

As others have pointed in their answers, bufio is probably what you want for reading from most files.

I'll add one other hint since it's really useful. Reading a line from a file is best accomplished not by the ReadLine method, but the ReadBytes or ReadString method instead.

Comments

0

ioutil.WriteFile and ReadFile are deprecated in Go 1.16.Even I could not find these methods in io package. I found those methods in os package. Following latest code works well.

package main

import "os"

func main() {
    os.WriteFile("sample.txt", []byte("Hello World"), 0666) //Writing file
    byteValue, _ := os.ReadFile("sample.txt")               //Reading file
    print(string(byteValue))
}

1 Comment

Just a note... if you have the tooling configured for your code editor, the warning that ioutil.ReadFile is deprecated also includes a recommendation of what function to use instead, in this case os.ReadFile

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.