Is it possible to return frozen, deduplicated strings from String#split function in Ruby?

You can get a frozen, deduplicated string via String#-@.

Either my using map:

str = '1,1,2,2'

str.split(',').map(&:-@).map(&:object_id)
#=> [70293234167580,
#    70293234167580,
#    70293368908400,
#    70293368908400]

or, using the block form to save memory while processing a huge string: (Ruby 2.6+)

def frozen_split(str, pattern)
  return enum_for(__method__, str, pattern) unless block_given?

  str.split(pattern) { |x| yield -x }
end

and call it via:

frozen_split(str, ',').map(&:object_id)
#=> [70293234167580,
#    70293234167580,
#    70293368908400,
#    70293368908400]

A simple to_sym also allows to reuse the same objects. For example:

puts "1,2,3,4,5,6".split(',').map(&:to_sym).map(&:object_id).inspect
puts "1,2,3,4,5,6".split(',').map(&:to_sym).map(&:object_id).inspect

This prints the same object ids:

[70236707757520, 70236707757480, 70236707757440, 70236707757400, 70236707757360, 70236707757320]
[70236707757520, 70236707757480, 70236707757440, 70236707757400, 70236707757360, 70236707757320]

Note that the to_sym method, as well as in the answer by Stefan, should save memory (I did not measure this), but the conversion itself does take some time.

So both methods that reuse object ids run slower than the default without conversion, see benchmarking results below (using ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-darwin18] ). Note that any code the uses these objects downstream could potentially run faster, but I was not sure what that code would be in your case.

Benchmarking code:

require 'benchmark' 

max_val = 10

[100, 1000, 10_000].each do |num_strings|
  puts "###############################"
  puts "num_strings=#{num_strings}:"
  puts "###############################"
  Benchmark.bmbm do |x|
    Kernel.srand(1234)
    x.report("default") { 10000.times { num_strings.times.map { rand(max_val) }.map(&:to_s).map(&:object_id) } }
    x.report("to_sym")  { 10000.times { num_strings.times.map { rand(max_val) }.map(&:to_s).map(&:to_sym).map(&:object_id) } }
    x.report("-@")      { 10000.times { num_strings.times.map { rand(max_val) }.map(&:to_s).map(&:-@).map(&:object_id) } }
  end
end

Benchmarking results:

###############################
num_strings=100:
###############################
Rehearsal -------------------------------------------
default   0.367201   0.000213   0.367414 (  0.367492)
to_sym    0.477524   0.000333   0.477857 (  0.478012)
-@        0.489703   0.000129   0.489832 (  0.489900)
---------------------------------- total: 1.335103sec

              user     system      total        real
default   0.369533   0.000336   0.369869 (  0.370126)
to_sym    0.504686   0.000775   0.505461 (  0.508025)
-@        0.497052   0.001251   0.498303 (  0.499578)
###############################
num_strings=1000:
###############################
Rehearsal -------------------------------------------
default   3.692454   0.005807   3.698261 (  3.706056)
to_sym    4.628710   0.003317   4.632027 (  4.633834)
-@        4.844655   0.004841   4.849496 (  4.865654)
--------------------------------- total: 13.179784sec

              user     system      total        real
default   3.583169   0.002604   3.585773 (  3.587418)
to_sym    4.709409   0.004160   4.713569 (  4.717487)
-@        4.909228   0.010225   4.919453 (  4.935606)
###############################
num_strings=10000:
###############################
Rehearsal -------------------------------------------
default  37.620197   0.117046  37.737243 ( 37.867851)
to_sym   48.576790   0.156409  48.733199 ( 48.948987)
-@       49.765026   0.105483  49.870509 ( 49.998702)
-------------------------------- total: 136.340951sec

              user     system      total        real
default  36.519696   0.068643  36.588339 ( 36.654737)
to_sym   47.571235   0.157084  47.728319 ( 47.937162)
-@       49.100705   0.177943  49.278648 ( 49.434869)

NOTE:

All these operations are pretty fast. It may be that the bottleneck in your case in not string allocations, etc, but the I/O: reading/writing large files. So you may need to optimize something entirely different, such as avoiding writing large files using pipes, etc.

Short Answer: No

If your goal is to use frozen strings to "save a lot of time and memory", then no, that can't be done with split, because Ruby isn't really designed for memory management concerns like that.

Basically:

long_string.split(',') # already too late, memory allocations have happened

But Possible

Your only recourse in pure Ruby is to not create the strings in the first place by implementing a streaming split yourself. Note that you'll have to avoid all the normal string iteration/access methods like each_char and even []:

str = "1,2,3,4,5"

# both will keep allocating new String objects
str.each_char.map(&:object_id)
(0...str.size).map { |i| str[i].object_id }

You'll have to use something like each_codepoint instead:

str.each_codepoint { |code| code } # does not keep allocating new objects

# so you could:
str.each_codepoint do |code|
  # implement your own parser, taking care to avoid dynamic memory allocations
end

In fact, if you are truly working with big files, you wouldn't even want to have the entire string loaded in memory at all. You'll want to stream file reads with something like IO.read

And just to get full closure, supposing you've implemented this, you can then monkeypatch that feature into String within your application to get what you wanted in the first place.

Proof

str = "1,2,3,4,5"
puts "Run in another shell:"
puts "watch -n 1 'ps ax -o pid,rss | grep -E \"^[[:space:]]*#{$$}\"'"
GC.disable

loop do
  # doesn't keep allocating memory
  str.each_codepoint { |code| code }

  # these keep allocating memory
  # str.each_char { |c| c }
  # (0...str.size).each { |i| str[i] }
end

Addendum

Extending from OP's PoC posted in another answer:

NUMS = [1, 2, 3]
LONG_STR = Array.new(99_999_999) { NUMS.sample.to_s }.join(','); nil

Benchmark.bm(20) do |m|
  m.report('my_split') { my_split(LONG_STR) }

  m.report('split') { LONG_STR.split(',') }

  results = [0, nil, nil, nil, nil, 0, 0, 0]
  m.report('tally w/o alloc') do
    LONG_STR.each_codepoint do |codepoint|
      results[codepoint - 44] += 1
    end
  end
end

# Run 1              user     system      total        real
# my_split        28.670430   0.541530  29.211960 ( 30.591287)
# split           11.633294   2.578581  14.211875 ( 14.561345)
# tally w/o alloc 12.797672   0.043086  12.840758 ( 12.963547)

# Run 2              user     system      total        real
# my_split        26.526297   0.897670  27.423967 ( 28.084112)
# split           23.000878   3.849396  26.850274 ( 28.269502)
# tally w/o alloc 12.919090   0.035687  12.954777 ( 13.196385)

Just FYI: benchmarking things where a lot of memory "thrashes" is always going to be pretty non-deterministic, as you've no control of when the garbage collector decides to kick in (and slows down execution).

Oh, and split might be even faster with #frozen_string_literal: true, and I've no idea what'd happen with --jit...

Thanks to Kache answer I drafted a PoC which solves my problem. That said, this code is much slower than the original split function.

COMMA_CODE_POINT = ','.ord
ONE_CODE_POINT = '1'.ord
TWO_CODE_POINT = '2'.ord
THREE_CODE_POINT = '3'.ord

def my_split(string)
  result = []
  current_string = []
  string.each_codepoint do |codepoint|
    if codepoint == COMMA_CODE_POINT
      process_string_part(current_string, result)
    else
      current_string << codepoint
    end
  end

  process_string_part(current_string, result)

  result
end

def process_string_part(current_string, result)
  if current_string.size == 1
    case current_string[0]
    when ONE_CODE_POINT
      result << -'1'
    when TWO_CODE_POINT
      result << -'2'
    when THREE_CODE_POINT
      result << -'3'
    else
      result << current_string.pack('U*')
    end
    current_string.clear
  elsif current_string.size > 0
    result << current_string.pack('U*')
    current_string.clear
  end
end

Here is a benchmark of this code:

a = "1,2,3,3,2,1,1,2,3,3,2,1,\\N,\\N,asdasda asdasd asdad"
n = 10_000_000

Benchmark.bmbm do |x|
  x.report("split") do
    n.times do
      a.split(',')
    end
  end
  x.report("my_split") do
    n.times do
      my_split(a)
    end
  end
end
            user     system      total        real
split    21.926568   0.000002  21.926570 ( 21.927100)
my_split 71.138833   0.000000  71.138833 ( 71.140378)

I was able to cut this time and get very close to the original implementation but with very limited functionality - the original string could only contain instances of the expected frozen strings and nothing else and frozen strings had to have one char only. I guess that in some cases this might be enough.