10

I work with massive strings which need a lot of manipulation.

For example, I might generate a string like this:

Part 1
Boat

Section A
Programming

Part 2
Partitioning boats for programming.

Section AA
Section SQL Entries.

The string would be too large to manually check every part of it. Now I need to split this string into a stringlist by sections and parts. I can think of two options:

A Regular Expression:

QStringList sl = s.split(QRegularExpression("\n(?=Part [0-9]+|Section [A-Z]+)"));

That looks like it should work, but sometimes exceptions slip through (IE: Section SQL Entries would erroneously get split)

Otherwise what I could do is place a marker when I generate the initial string:

🚤💻Part 1
Boat

🚤💻Section A
Programming

🚤💻Part 2
Partitioning boats for programming.

🚤💻Section AA
Section SQL Entries.

Which means that splitting the string would become easy:

QStringList sl = s.split("🚤💻"));

Something tells me though that neither of these are good style or programming practice, but I have up until this point not discussed it nor found an alternative.

  • If you were my project manager, would you accept either of these methods?
  • If not, what would you suggest I do as a best practice?
Improve this question
7
  • 6
    If your program knows where to place these markers, why not generate the sections as separate strings to begin with? Commented Nov 21, 2016 at 12:37
  • I don't think user a marker that doesn't translate well into your current encoding is a good idea. Commented Nov 21, 2016 at 12:39
  • 2
    the actual symbols used are largely irrelevant, what is going to make a difference is the grammar of the the thing you are trying to parse Commented Nov 21, 2016 at 13:05
  • 4
    @Akiva are you sure about the performance hit? You're working with the same amount of data in any case, I doubt there would be a significant difference. Compose the thousands of functions into one function, invoke that in a loop, and take some measurements. Commented Nov 21, 2016 at 13:14
  • 2
    @Akiva Retrieving and replacing elements in a list should at worst be comparable to splitting a big string. Commented Nov 21, 2016 at 16:27

5 Answers 5

Reset to default
17

It's not bad practice to have document encoding embedded as text in a string. Think of markdown, HTML, XML, JSON, YAML, LaTeX, etc.

What is bad practice is reinventing the wheel. Rather than writing your own text processor, think about using an existing standard. There's plenty of free software that do much of the parsing for you, and many have a non-restrictive license that let you use said software in your own proprietary software.

Improve this answer
8
  • In my case, I am inventing a wheel, if what I am trying to do is build a unique interpreter for a markdown language. For example, one of my projects was interpreting Latex as SSML that is readable by the human ear: meta.wikimedia.org/wiki/Grants:IdeaLab/…. << There is a period at the end of that URL, otherwise it won't work Commented Nov 21, 2016 at 13:40
  • 2
    @Akiva I have to work with a custom text format developed by my workplace that literally reinvents the wheel. I have to maintain 4 parsers in 3 languages (Javascript, Java and Objective-C) for it, and it's a fricking nightmare. Do the right thing now and abolish this custom text format nonsense. I cannot stress enough how huge of a maintenance nightmare this will become a few years down the road. Use existing structured formats, XML, JSON, etc. Commented Nov 24, 2016 at 17:55
  • @ChrisCirefice Can you give me an example of how it is a nightmare? Commented Nov 24, 2016 at 18:25
  • 1
    @Akiva I think the fact that you have to maintain even one parser (in my case several and in different languages) is horrid. Standard formats exist for a reason - they can represent the data you need them to - and with extremely little effort on your part, because those parsers have been built, refined and are maintained. The custom text format is also extremely specialized knowledge, meaning that usually only one or two developers will be familiar enough with the format to successfully maintain it. That should speak volumes. Most people are familiar with CML, JSON - few know custom formats. Commented Nov 24, 2016 at 23:53
  • 1
    @Akiva Indeed! Markdown format (what SE and many other sites use for text formatting) is somewhat standard, like SQL is. But there are many different 'flavors' with custom extensions (e.g. like SE). There is a standard library that parses the 'core', then you extend the library if you want additional features. But, building and maintaining your own formatter would be ludicrous - several already exist (markdown, BB code, etc.), so why reinvent the wheel and maintain all that code? May as well just use an existing library :) Commented Nov 26, 2016 at 15:49
8

Using some common separator should work fine when splitting larger arbitrary strings, but I would recommend against using an arbitrary symbol. Someone reading that string as plaintext could be confused, not to mention troubles with UTF and whether or not the symbol appears inside the sections or not.

The most important part of this is that each section remains intact, while each "section header" needs to be appropriately identified.

Why not use a common separator but keep it readable? Something like:

[SECTION]
Part 1
Boat

[SECTION]
Section A
Programming

[SECTION]
Part 2
Partitioning boats for programming.

[SECTION]
Section AA
Section SQL Entries.

The problem is deciding what the separator should be, as it needs to be something that is guaranteed to not show up any section. You could further identify it as a separator by requiring it is at the start of a line and the only text on that line.

Without further knowledge of what text is expected in each section it's hard to make a recommendation on what common separator would be best in this case.

Improve this answer
1
  • I like your answer's emphasis on readability. The strings are generated through data scraping user generated text, say for example, the Markup language used in SE for writing questions and answers. Thus you could easily imagine what type of string manipulation issues could come into play. Commented Nov 21, 2016 at 13:22
5

The accepted answer seems to have missed what you wrote in a comment:

The reason is that a lot of the manipulation I do requires the full string

and gave this as an example:

s.replace("boat", "programming");

If that is what you want, it is IMHO a really bad idea to use some "markdown" or textual separator for your whole string, this has always a certain risk to interfer with the manipulation and will not lead to robust code. Especially when you try to start using regular expressions on such a combined string, you will probably run into the same problems people observed when trying to parse HTLM or XML with regular expressions.

Especially because you wrote there might be "thousands of [such manipulation] functions", that risk might become a real problem. Even if you use some markdown like XML to store the string list internally, you need to make sure the manipulation will process only the content, not the markdown, so that would mean to split the string into parts before you do any processing, and join it afterwards again - so that will have a high risk of giving you a bad performance.

The better design alternative here is to provide an abstract datatype (use a class if you like), lets call it MyStringList, and provide a small set of basic operations which allow you to implement your "thousands of functions" in terms of that operations. For example, there might be generic find and replace operations, or a generic functional map operation. You can also add something like a JoinToString operation if you really need the whole list in one string for certain purporses.

Using these operations, your fear that the code becomes more complicated because "everything would have to be done in a for loop" becomes pointless, because the only for loops you get are encapsulated inside the datatype's operations. And I would not be concerned about the performance until you have a real, measureable performance impact (which I doubt you get if you implement the basic operations correctly).

Improve this answer
1
  • Upvote because I did actually create something like that. It allows me to set custom brackets say, < and >, and it will grab every instance of that string where I can easily remove the instances I do not want, and cleanly manipulate it in the way I want. This is good because regular expressions by themselves do not handle substrings like this: <boat <programming>> well where there are multiple layers of brackets. Commented Nov 22, 2016 at 14:34
1

The format that is described is very similar to INI files:

https://en.wikipedia.org/wiki/INI_file

In that case the section is enclosed by square brackets [] so what you describe makes sense by marking the section in some fashion to add additional meaning to that text.

Improve this answer
0

For example, I might generate a string like this:

Question: From what do you "generate" this string?

Would that be any easier to manipulate?

Improve this answer
3
  • String is generated from Datascraping user content from a website. Commented Nov 22, 2016 at 14:04
  • 1
    This is not a reliable way to retrieve data from a web site, simply because they change and things get moved around or disappear entirely. You would be far better off retrieving the data from some sort of published (and therefore reliable) API. Furthermore, the use of many commercial web sites specifically bans this sort of thing. Commented Nov 22, 2016 at 14:49
  • Sometimes I do not get to choose what data is valuable to me, and so there is always a need to do integrity checks for what you are looking at, or just plain compromise and hope for the best. For example: I wrote a LaTeX to SSML interpreter, and one of the issues is that you can generate identical images with vastly different code, and so it is nigh impossible to be consistent if the user chooses poor or esoteric ways of generating his formulas. All that means at the end of the day is that people who do not use good practice will not have a decent interpretations of their scripts. Commented Nov 22, 2016 at 17:27

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.