Customize checkbox checked style

You must use the label, because otherwise you cannot check the checkbox when you click on it.

For the "Remember me", you can add another label, which contains the text. To have it on the same line, you must add display: inline-block to #wrap

<div id="wrap">
    <input type='checkbox' name='thing' value='valuable' id="thing" />
    <label class="mark" for="thing"></label>
</div>
<label for="thing">Remember me</label>

#wrap {
    display: inline-block;
}

To be more like the native checkbox when unchecked, you can use

background: transparent;

To distinguish between the two labels, I used class="mark" for the box.

See modified JSFiddle

This works in Chrom, but at least for Firefox 20

#thing {
    display: none;
}

doesn't work. For Firefox 20, you must use

#thing {
    position: absolute;
    visibility: hidden;
}

to hide the checkbox and take it out of the normal flow.

JSFiddle

Update:

You can also try #thing + label:before to create a checkbox

#thing + label:before
{
    content: "";
    background: transparent;
    height: 9px;
    width: 9px;
    display:inline-block;
    padding: 0 0 0 0px;
    margin-left: 3px;
    margin-bottom: 3px
}
#thing:checked + label:before
{
    background: #0080FF;
}

and put the "Remember me" in the label

<input type='checkbox' name='thing' value='valuable' id="thing"/>
<label for="thing">Remember me</label>

The drawback to this, I cannot find a way to put the box and label on one line.

JSFiddle