1

OS: CentOS 5.10
Service: Amazon Web Services
Attempted to remove MySQL 5.0 and install MySQL 5.6

I had a server set up, and was going to use MySQL, but it was was set up with MySQL 5.0. Since there are performance improvements and other features added, and there wasn't yet any data or anything, I wanted to install MySQL 5.6.

I was following the guide posted by Oracle to get an RPM installed for use on RHEL 5:

http://dev.mysql.com/doc/mysql-repo-excerpt/5.6/en/linux-installation-yum-repo.html

Everything went swimmingly, and I could start the service immediately after installing - but...

When I got to this step:

Always run the program mysql_secure_installation to secure your MySQL installation:

shell> mysql_secure_installation

mysql_secure_installation allows you to perform important operations like setting root password, removing anonymous users, and so on. The program is safe and easy to use. It is important to remember the root password you set though. See mysql_secure_installation — Improve MySQL Installation Security for details.

I ran mysql_secure_installation, and when it asked for a password, I attempted to just press enter, since I had not yet set one for root@localhost.

I was informed that the password was incorrect.

So tried what the previous password was, and was given an error that I have since lost because I'm connecting through putty, and the window closed when I lost the connection.

In any case - I just asked for the password again, so I tried just pressing enter, and the old password again, niether worked.

So I cancelled the process, ran sudo /sbin/service mysqld stop and then sudo /sbin/service mysqld start. The shell stopped, hung for a second, and then the window closed, and thus my connection.

I tried to open the ssh connection again, but it timed out. After waiting a few minutes, I could ssh back into the server.

Every time I attempt to start the mysqld service, my ssh session closes. Does that maybe mean that sshd is crashing as well?

The log file /var/log/mysql.log contains only the following:

131204 23:49:02 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
131204 23:50:53 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
131205 00:05:12 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
131205 00:06:28 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
131205 00:08:09 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
131205 00:09:14 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
131205 00:13:54 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
131205 00:15:04 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
131205 00:25:12 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
131205 00:26:45 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

I've tried running mysql manually, and the same thing happens.

What do I do from here?

Edit:

I just ran mysqld_safe to see what the output is, and it gets this far:

[user@machine mysql]$ sudo /usr/bin/mysqld_safe
131205 01:13:33 mysqld_safe Logging to '/var/log/mysqld.log'.
131205 01:13:33 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Killed

Before hanging up on me.

What happens if I move those databases that InnoDB uses?

As per Patrick, I checked dmesg, and searched for mysqld.

I found the following:

type=1400 audit(1386200078.706:1929): avc:  denied  { read } for  pid=20563 comm="nohup" path="/usr/sbin/mysqld" dev=xvda1 ino=375035 scontext=user_u:system_r:netutils_t:s0 tcontext=system_u:object_r:mysqld_exec_t:s0 tclass=file
sudo invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0

Call Trace:
 [<ffffffff802c1b64>] out_of_memory+0x8b/0x203
 [<ffffffff8020fa5d>] __alloc_pages+0x27f/0x308
 [<ffffffff802139dd>] __do_page_cache_readahead+0xc8/0x1af
 [<ffffffff8021424e>] filemap_nopage+0x14c/0x360
 [<ffffffff80208e9d>] __handle_mm_fault+0x444/0x144f
 [<ffffffff8026ea01>] monotonic_clock+0x35/0x7b
 [<ffffffff8029b54a>] attach_pid+0x7c/0xa9
 [<ffffffff80266d94>] do_page_fault+0xf72/0x131b
 [<ffffffff8024901b>] skb_dequeue+0x48/0x50
 [<ffffffff80254146>] unix_release_sock+0x19e/0x1fa
 [<ffffffff80261df5>] thread_return+0x6c/0x113
 [<ffffffff80207116>] kmem_cache_free+0x84/0xd7
 [<ffffffff80207116>] kmem_cache_free+0x84/0xd7
 [<ffffffff8025f82b>] error_exit+0x0/0x6e

And:

mysqld invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0

Call Trace:
 [<ffffffff802c1b64>] out_of_memory+0x8b/0x203
 [<ffffffff8020fa5d>] __alloc_pages+0x27f/0x308
 [<ffffffff802139dd>] __do_page_cache_readahead+0xc8/0x1af
 [<ffffffff8021424e>] filemap_nopage+0x14c/0x360
 [<ffffffff80208e9d>] __handle_mm_fault+0x444/0x144f
 [<ffffffff8020622a>] hypercall_page+0x22a/0x1000
 [<ffffffff8020622a>] hypercall_page+0x22a/0x1000
 [<ffffffff80266d94>] do_page_fault+0xf72/0x131b
 [<ffffffff80236fd3>] __vm_enough_memory+0xee/0xfc
 [<ffffffff8020e472>] do_mmap_pgoff+0x35e/0x74a
 [<ffffffff80236fd3>] __vm_enough_memory+0xee/0xfc
 [<ffffffff80263929>] _spin_lock_irqsave+0x9/0x14
 [<ffffffff802313f1>] __up_write+0x27/0xf2
 [<ffffffff8025f82b>] error_exit+0x0/0x6e

Mem-info:
DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: high 186, batch 31 used:79
cpu 0 cold: high 62, batch 15 used:58
Normal per-cpu: empty
HighMem per-cpu: empty
Free pages:        4780kB (0kB HighMem)
Active:114881 inactive:1 dirty:1 writeback:0 unstable:0 free:1195 slab:5226 mapped-file:541 mapped-anon:114329 pagetables:1094
DMA free:2004kB min:48kB low:60kB high:72kB active:540kB inactive:0kB present:9076kB pages_scanned:1381592 all_unreclaimable? yes
lowmem_reserve[]: 0 489 489 489
DMA32 free:2776kB min:2804kB low:3504kB high:4204kB active:458984kB inactive:4kB present:500960kB pages_scanned:10643850 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2004kB
DMA32: 28*4kB 1*8kB 0*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2776kB
Normal: empty
HighMem: empty
585 pagecache pages
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 0kB
Total swap = 0kB
Free swap:            0kB
131072 pages of RAM
5993 reserved pages
4445 pages shared
0 pages swap cached
Out of memory: Killed process 20563, UID 0, (mysqld).

What do I do from here?

Improve this question
3
  • Some googling brought up this page that you might find helpful: dev.mysql.com/doc/refman/5.5/en/crashing.html Commented Dec 5, 2013 at 1:23
  • 2
    Does dmesg show anything interesting? Kernel OOM messages perhaps? I can see 2 things happening. 1) the process is consuming all the memory upon which you get an OOM condition and the kernel tries to recover by killing procs, and so it kills your shell. Or 2) the system is kernel panicing and rebooting. If you're using amazon's t1.micro instance, the OOM is very likely. Commented Dec 5, 2013 at 1:38
  • @Patrick! I think you were right! I added the information to the post. Commented Dec 5, 2013 at 1:49

2 Answers 2

Reset to default
5

From the dmesg output you provided, it is indeed the OOM killer that is causing your problem.
Basically what happens is that upon launching mysql, the system exhausts all its available memory. When that happens the kernel has to kill something to get some back. It has a complex algorithm it uses to try and find something to kill that won't cause system instability. The algorithm happens to choose your shell. Then after your shell is killed, it finds that wasn't enough and starts killing other things. Eventually it gets to the mysql process and kills it.

The dmesg output also shows you only have about 500mb of memory on the system. I would normally think this is enough to run mysql if it's small, and if you don't care about performance. But obviously this indicates otherwise.
Is there anything else running on the box using a lot of memory? The dmesg output should have also included a list of processes and how much memory they were consuming.

If there's nothing else using an abnormal amount of memory, there are 2 options:

  1. Enable swap
  2. Add more memory (use a larger instance size).
Improve this answer
4
  • Thank you, Patrick. Are you referring to the "present:500960kB"? I did check free -m, and indeed it said there are 512 megs of memory. I'm taking a look through the list of processes - but unfortunately I'm unfamiliar with the output - how do I tell how much memory the process was using when it was killed? I watched top on a separate shell, and when I start mysqld, it immediately uses 21% of the available memory. I don't see anything else that's taking up a lot. Commented Dec 5, 2013 at 2:30
  • Yes, the present:500960kB is what told me how much memory you have. In the process list that gets dumped into dmesg at the time of the OOM condition, one of the columns should include how much memory each process is using. If you can put that output in your question we can take a look at it and see if there's anything obviously wrong. But it very well could be just mysqld. If mysql is performing some memory intensive operation on start, then you're just kinda stuck :-/ Commented Dec 5, 2013 at 2:40
  • @Zaemz With MySQL you typically want to allocate 80% of your RAM to the innodb_buffer_pool. This should be about 10% larger than the size of your data for optimal performance. In other words, databases should have tons of RAM (or SSDs, but RAM is much cheaper). Commented Dec 5, 2013 at 3:02
  • Thanks for the information @jordanm. I eventually did what Patrick said to do, and that was enable the swap. This fixed my issue. Commented Dec 6, 2013 at 0:23
-1

Sounds like your old database files from the existing mysql installation is still hanging around. If there is data you need to retain run the mysql_upgrade command. If you don't need the existing data wipe out /var/lib/mysql and re-run the mysql_secure_installation script.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.