Skip to main content
Add implementation using `std::views::split`
Source Link
tom
  • 23.2k
  • 7
  • 47
  • 38

Because there is no operator >> on std::stringstream and std::string_view (and std::istream_iterator requires this operator).

As @tkausl points out in the comments, it's not possible for >> to work on std::string_view because it's not clear who would own the memory pointed to by the std::string_view.

In the case of your program, ss << inputStr copies the characters from inputStr into ss, and when ss goes out of scope its memory would be freed.


Here is a possible implementation that uses C++20's std::ranges::views::split instead of std::stringstream. It only supports a single space as the delimiter.

#include <iostream>
#include <sstream>
#include <string>
#include <string_view>
#include <vector>
#include <iterator>
#include <algorithm>
#include <ranges>


[[ nodiscard ]] std::vector< std::string_view >
tokenize( const std::string_view inputStr, const size_t expectedTokenCount )
{
    constexpr std::string_view delim { " " };

    std::vector< std::string_view > foundTokens { };

    if ( inputStr.empty( ) ) [[ unlikely ]]
    {
        return foundTokens;
    }

    foundTokens.reserve( expectedTokenCount );
    for ( const auto token : std::views::split( inputStr, delim ) )
    {
        foundTokens.emplace_back( token.begin( ), token.end( ) );
    }

    return foundTokens;
}

int main( )
{
    using std::string_view_literals::operator""sv;
    constexpr auto text { "Today is a nice day."sv };

    const auto tokens { tokenize( text, 4 ) };

    std::cout << tokens.size( ) << '\n';
    std::ranges::copy( tokens, std::ostream_iterator< std::string_view >{ std::cout, "\n" } );
}

This works with gcc 12.1 (compile with -std=c++20), but it doesn't work with clang 14.0.0 because clang hasn't implemented P2210 yet.

Because there is no operator >> on std::stringstream and std::string_view (and std::istream_iterator requires this operator).

As @tkausl points out in the comments, it's not possible for >> to work on std::string_view because it's not clear who would own the memory pointed to by the std::string_view.

In the case of your program, ss << inputStr copies the characters from inputStr into ss, and when ss goes out of scope its memory would be freed.

Because there is no operator >> on std::stringstream and std::string_view (and std::istream_iterator requires this operator).

As @tkausl points out in the comments, it's not possible for >> to work on std::string_view because it's not clear who would own the memory pointed to by the std::string_view.

In the case of your program, ss << inputStr copies the characters from inputStr into ss, and when ss goes out of scope its memory would be freed.


Here is a possible implementation that uses C++20's std::ranges::views::split instead of std::stringstream. It only supports a single space as the delimiter.

#include <iostream>
#include <sstream>
#include <string>
#include <string_view>
#include <vector>
#include <iterator>
#include <algorithm>
#include <ranges>


[[ nodiscard ]] std::vector< std::string_view >
tokenize( const std::string_view inputStr, const size_t expectedTokenCount )
{
    constexpr std::string_view delim { " " };

    std::vector< std::string_view > foundTokens { };

    if ( inputStr.empty( ) ) [[ unlikely ]]
    {
        return foundTokens;
    }

    foundTokens.reserve( expectedTokenCount );
    for ( const auto token : std::views::split( inputStr, delim ) )
    {
        foundTokens.emplace_back( token.begin( ), token.end( ) );
    }

    return foundTokens;
}

int main( )
{
    using std::string_view_literals::operator""sv;
    constexpr auto text { "Today is a nice day."sv };

    const auto tokens { tokenize( text, 4 ) };

    std::cout << tokens.size( ) << '\n';
    std::ranges::copy( tokens, std::ostream_iterator< std::string_view >{ std::cout, "\n" } );
}

This works with gcc 12.1 (compile with -std=c++20), but it doesn't work with clang 14.0.0 because clang hasn't implemented P2210 yet.

Technically it *is* to do with istream_iterator, because istream_iterator tries to call that operator.
Source Link
tom
  • 23.2k
  • 7
  • 47
  • 38

Because there is no operator >> on std::stringstream and std::string_view (nothing to do withand std::istream_iterator requires this operator).

As @tkausl points out in the comments, it's not possible for >> to work on std::string_view because it's not clear who would own the memory pointed to by the std::string_view.

In the case of your program, ss << inputStr copies the characters from inputStr into ss, and when ss goes out of scope its memory would be freed.

Because there is no operator >> on std::stringstream and std::string_view (nothing to do with std::istream_iterator).

As @tkausl points out in the comments, it's not possible for >> to work on std::string_view because it's not clear who would own the memory pointed to by the std::string_view.

In the case of your program, ss << inputStr copies the characters from inputStr into ss, and when ss goes out of scope its memory would be freed.

Because there is no operator >> on std::stringstream and std::string_view (and std::istream_iterator requires this operator).

As @tkausl points out in the comments, it's not possible for >> to work on std::string_view because it's not clear who would own the memory pointed to by the std::string_view.

In the case of your program, ss << inputStr copies the characters from inputStr into ss, and when ss goes out of scope its memory would be freed.

Source Link
tom
  • 23.2k
  • 7
  • 47
  • 38

Because there is no operator >> on std::stringstream and std::string_view (nothing to do with std::istream_iterator).

As @tkausl points out in the comments, it's not possible for >> to work on std::string_view because it's not clear who would own the memory pointed to by the std::string_view.

In the case of your program, ss << inputStr copies the characters from inputStr into ss, and when ss goes out of scope its memory would be freed.