1

With Buildroot I'm making images for my Embedded Linux Hardware. Mainly I'm trying to speed up the boot sequence (and on the way lower the memory usage), where I've tried many techniques successfully.

What I'd like to do: Recently I've heared about removing duplicated files in a directory (by e.g. replacing those files with symbolic links) and I'd like to apply this method to my rootfs

The surroundings: With Buildroot I can have many different types of rootfs formats (cramfs, cpio, ext2/3/4, etc.), which are created during make as 1 (packed) file (e.g.: rootfs.cpio). Now I don't really know how to:

  • open up the image
  • remove duplicated files (well I know how to remove duplicated files in general)
  • pack the rootfs again - so that I still can use it to flash and execute it on my hardware

Maybe it's not even possible at all. I believe at least when using static libraries, many files can be replaced.

Has somebody an idea?

Improve this question
5
  • 1
    I compare the md5sum of files to find duplicates. I don't know much about embedded systems, but I know that specialized filesystems such as squashfs do this kind of deduplication automatically. You might want to consider "hard" links instead of symbolic ones in case of picky software that expects to find a real binary. Commented Sep 16, 2014 at 7:06
  • I'm surprised that you have duplicated files in the first place. What are they? This smacks of a broken build script. Commented Sep 16, 2014 at 21:24
  • @Gilles well I haven't checked yet, that's what I would like to do. But as mentioned in the top post, I guess at least some libraries can be replaced. I won't know it if I won't test it Commented Sep 17, 2014 at 6:04
  • I don't understand why you think some libraries will be duplicated. If you link a library statically, some bits of code will be duplicated, but not files, and not even disk blocks either because the same code will show up at different alignments and possibly with different bits inlined. If you link a library dynamically, the code will only appear once, in the single dynamic library file. Commented Sep 17, 2014 at 6:11
  • Well I get your point, but I don't know what happens if you're including additional Packages into your rootfs. E.g.: if you have multiple crypto programs, i could imagine that everyone of those will come with their own libraries. Or e.g. if you let the user create packages/programs that will be build into the rootfs, so it is really unknown what's build twice and what is not. Commented Sep 17, 2014 at 6:17

2 Answers 2

Reset to default
2

I can't help with the buildroot specific stuff (other than to say that you'd have to either mount it or boot it before removing duplicates, and then creating a new rootfs from the resulting filesystem), but removing duplicates is quite possible. A naive approach would be to find / -type f -exec sha256sum {} + | sort > checksums.txt, read the resulting file line by line with while read -r checksum path, compare each checksum to the previous, and deleting (or hardlinking) the duplicates. This wouldn't handle weird filenames, for example containing newlines, but that shouldn't be a problem in your case.

Improve this answer
3
  • ... does this also fetch a rootfs compressed to 1 file, like e.g. rootfs.ext4 ? Commented Sep 16, 2014 at 10:53
  • I'm afraid I don't understand the question. Commented Sep 16, 2014 at 15:06
  • if you're using a build system (like buildroot) then it will create you a rootfs in/as a single file, depending on what you've chosen you'll get e.g. a rootfs.ext4. Now this file can be used to flash to a partition on a SD card with the dd command - and then you have the typical linux file system structure with the home, usr, etc ... directories - so you can configure linux to read the rootfs from the SD-Card partition, which is working pretty fine. Commented Sep 17, 2014 at 6:06
1

I don't see how static libraries enter into the mix. Static libraries are not present on the target root filesystem with Buildroot.

Generally speaking, Buildroot tries to install only what's necessary on the target, and I honestly doubt you will find many duplicated files there.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.