detect non ascii characters in a string

Question

How can I detect non-ascii characters in a vector of strings in a grep like fashion. For example below I'd like to return c(1, 3) or c(TRUE, FALSE, TRUE, FALSE):

x <- c("façile test of showNonASCII(): details{", 
    "This is a good line", "This has an ümlaut in it.", "OK again. }")

Attempt:

y <- tools::showNonASCII(x)
str(y)
p <- capture.output(tools::showNonASCII(x))

@David I think that will do it... can you throw down as an answer. Maybe others will see an issue with it or have different solutions. — Tyler Rinker
– Tyler Rinker, Commented Jan 5, 2016 at 14:23
Why not fix the code so it handles Unicode properly instead? — Panagiotis Kanavos
– Panagiotis Kanavos, Commented Jan 5, 2016 at 14:26
@PanagiotisKanavos I will, that's easy, but this is to validate strings so I first need to detect if there's a problem with the data so as to inform the client. — Tyler Rinker
– Tyler Rinker, Commented Jan 5, 2016 at 14:29
@PanagiotisKanavos b/c it's data from a client. We want it in a particular format. Non-standard data format is a data scientist's enemy, particularly if you're trying to automate a task. It's far easier and cheaper to get clients to put data in the correct format than to try to clean up and address un-foreseen errors later. — Tyler Rinker
– Tyler Rinker, Commented Jan 5, 2016 at 14:42

Tyler Rinker · Accepted Answer · 2016-01-15 03:11:26Z

31

Came across this later using pure base regex and so simple:

grepl("[^ -~]", x)
## [1]  TRUE FALSE  TRUE FALSE

More here: http://www.catonmat.net/blog/my-favorite-regex/

answered Jan 15, 2016 at 3:11

Tyler Rinker

111k74 gold badges335 silver badges533 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

sbha Over a year ago

From the link: "[ -~] matches all ASCII characters from the space to tilde. What are these characters? These are all printable characters!"

Laurent Bergé Over a year ago

Short, simple, smart: simply beautiful!

David Arenburg · Accepted Answer · 2016-01-05 14:48:59Z

22

another possible way is to try to convert your string to ASCII and the try to detect all the generated non printable control characters which couldn't be converted

grepl("[[:cntrl:]]", stringi::stri_enc_toascii(x))
## [1]  TRUE FALSE  TRUE FALSE

Though it seems stringi has a built in function for this type of things too

stringi::stri_enc_mark(x)
# [1] "latin1" "ASCII"  "latin1" "ASCII"

answered Jan 5, 2016 at 14:48

David Arenburg

92.4k18 gold badges143 silver badges201 bronze badges

1 Comment

Tyler Rinker Over a year ago

Both solutions are terrific. This one is a bit more compact and may be more robust to other encodings, though, admittedly, I know very little about encodings.

Roland · Accepted Answer · 2016-01-05 14:47:40Z

13

Why don't you extract the relevant code from showNonASCII?

x <- c("façile test of showNonASCII(): details{", 
       "This is a good line", "This has an ümlaut in it.", "OK again. }")

grepNonASCII <- function(x) {
  asc <- iconv(x, "latin1", "ASCII")
  ind <- is.na(asc) | asc != x
  which(ind)
}

grepNonASCII(x)
#[1] 1 3

answered Jan 5, 2016 at 14:47

Roland

134k12 gold badges203 silver badges305 bronze badges

4 Comments

Heatshock Over a year ago

The iconv function appears to remove the variable label attributes randomly in a dataframe when applied. What could be reasons?

Roland Over a year ago

@Heatshock I have no idea what you are doing. iconv should preserve attributes.

Heatshock Over a year ago

I read in a sas dataset in xpt format. here is code. dv <-read_xpt('adsl.xpt'). dv1<-dv |> mutate(across(everything(), ~iconv(.,"latin2", "ascii") )) then use str(dv1). Some of the variable labels get lost in a consistent way. but not all variables. Do you know why?

Roland Over a year ago

No, I don't. Might be due to your use of dplyr.

Odin · Accepted Answer · 2019-07-30 13:55:13Z

8

A bit late I guess but it could be useful for the next readers.

You can find these functions:

showNonASCII(<character_vector>)
showNonASCIIfile(<file>)

in the tools R package (see https://stat.ethz.ch/R-manual/R-devel/library/tools/html/showNonASCII.html). It does exactly what is asked here: show non ASCII characters in a string or in a text file.

answered Jul 30, 2019 at 13:55

Odin

7155 silver badges18 bronze badges

Comments

Wiktor Stribiżew · Accepted Answer · 2021-10-07 15:10:02Z

A stringr regex solution:

library(stringr)
x <- c("façile test of showNonASCII(): details{", 
    "This is a good line",
    "This has an ümlaut in it.", "OK again. }")
str_detect(x, "[^[:ascii:]]")
# => [1]  TRUE FALSE  TRUE FALSE

The [^[:ascii:]] pattern matches any non-ASCII character.

The [[:ascii:]] pattern matches any ASCII character.

If you ever need to make sure the whole string consists of non-ASCII chars, use

str_detect(x, "^[^[:ascii:]]+\\z")

where ^ matches the start of string and \z matches the very end of string.

Collectives™ on Stack Overflow

detect non ascii characters in a string

5 Answers 5

2 Comments

1 Comment

4 Comments

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

2 Comments

1 Comment

4 Comments

Comments

Comments

Linked

Related