Best way to simulate "group by" from bash?

sort ip_addresses | uniq -c

This will print the count first, but other than that it should be exactly what you want.

The quick and dirty method is as follows:

cat ip_addresses | sort -n | uniq -c

If you need to use the values in bash you can assign the whole command to a bash variable and then loop through the results.

PS

If the sort command is omitted, you will not get the correct results as uniq only looks at successive identical lines.

The canonical solution is the one mentioned by another respondent:

sort | uniq -c

It is shorter and more concise than what can be written in Perl or awk.

You write that you don't want to use sort, because the data's size is larger than the machine's main memory size. Don't underestimate the implementation quality of the Unix sort command. Sort was used to handle very large volumes of data (think the original AT&T's billing data) on machines with 128k (that's 131,072 bytes) of memory (PDP-11). When sort encounters more data than a preset limit (often tuned close to the size of the machine's main memory) it sorts the data it has read in main memory and writes it into a temporary file. It then repeats the action with the next chunks of data. Finally, it performs a merge sort on those intermediate files. This allows sort to work on data many times larger than the machine's main memory.

It seems that you have to either use a big amount of code to simulate hashes in bash to get linear behavior or stick to the quadratic superlinear versions.

Among those versions, saua's solution is the best (and simplest):

sort -n ip_addresses.txt | uniq -c

I found http://unix.derkeiler.com/Newsgroups/comp.unix.shell/2005-11/0118.html. But it's ugly as hell...

You probably can use the file system itself as a hash table. Pseudo-code as follows:

for every entry in the ip address file; do
  let addr denote the ip address;

  if file "addr" does not exist; then
    create file "addr";
    write a number "0" in the file;
  else 
    read the number from "addr";
    increase the number by 1 and write it back;
  fi
done

In the end, all you need to do is to traverse all the files and print the file names and numbers in them. Alternatively, instead of keeping a count, you could append a space or a newline each time to the file, and in the end just look at the file size in bytes.

I feel awk associative array is also handy in this case

$ awk '{count[$1]++}END{for(j in count) print j,count[j]}' ips.txt

A group by post here

I'd have done it like this:

perl -e 'while (<>) {chop; $h{$_}++;} for $k (keys %h) {print "$k $h{$k}\n";}' ip_addresses

but uniq might work for you.

I understand you are looking for something in Bash, but in case someone else might be looking for something in Python, you might want to consider this:

mySet = set()
for line in open("ip_address_file.txt"):
     line = line.rstrip()
     mySet.add(line)

As values in the set are unique by default and Python is pretty good at this stuff, you might win something here. I haven't tested the code, so it might be bugged, but this might get you there. And if you want to count occurrences, using a dict instead of a set is easy to implement.

Edit: I'm a lousy reader, so I answered wrong. Here's a snippet with a dict that would count occurences.

mydict = {}
for line in open("ip_address_file.txt"):
    line = line.rstrip()
    if line in mydict:
        mydict[line] += 1
    else:
        mydict[line] = 1

The dictionary mydict now holds a list of unique IP's as keys and the amount of times they occurred as their values.

This does not answer the count element of the original question, but this question is the first search engine result when searching for what I wanted to achieve, so I thought this may help someone as it relates to 'group by' functionality.

I wanted to order files based on groupings of them, where the presence of some string in the filename determined the group.

It uses a temporary grouping/ordering prefix which is removed after ordering; sed substitute expressions (s#pattern#replacement#g) match the target string and prepend an integer to the line corresponding to the desired sort order of that target string. Then, grouping prefix is removed with cut.

Note that the sed expressions could be joined (e.g. sed -e '<expr>; <expr>; <expr>;') but here they're split for readability.

It's not pretty and probably not fast (I'm dealing with <50 items) but it at-least conceptually simple and doesn't require learning awk.

#!/usr/bin/env bash

for line in $(find /etc \
    | sed -E -e "s#^(.*${target_string_A}.*)#${target_string_A_sort_index}:\1#;" \
    | sed -E -e "s#^(.*${target_string_B}.*)#${target_string_B_sort_index}:\1#;" \
    | sed -E -e "s#^/(.*)#00:/\1#;" \
    | sort \
    | cut -c4-
)
do
    echo "${line}"
done

e.g. Input

/this/is/a/test/a
/this/is/a/test/b
/this/is/a/test/c
/this/is/a/special/test/d
/this/is/a/another/test/e

#!/usr/bin/env bash

for line in $(find /etc \
    | sed -E -e "s#^(.*special.*)#10:\1#;" \
    | sed -E -e "s#^(.*another.*)#05:\1#;" \
    | sed -E -e "s#^/(.*)#00:/\1#;" \
    | sort \
    | cut -c4-
)
do
    echo "${line}"
done

/this/is/a/test/a
/this/is/a/test/b
/this/is/a/test/c
/this/is/a/another/test/e
/this/is/a/special/test/d

Importing data to sqlite db and using sql syntax (just an other idea). I know it's too much for this example but would be useful for complex queries with multiple files (tables)

#!/bin/bash
trap clear_db EXIT
clear_db(){ rm -f "mydb$$"; }

# add header to input_file (IP)
INPUT_FILE=ips.txt

# import file into db
sqlite3 -csv mydb$$ ".import ${INPUT_FILE} mytable"

# using sql statements on table 'mytable' 
sqlite3 mydb$$ -separator " "  "SELECT IP, COUNT(*) FROM mytable GROUP BY IP;"

10.0.10.1 3
10.0.10.2 1
10.0.10.3 1

A combination of awk + sort (with version sort flag) is probably fastest (if ur environment has awk at all):

echo "${input...}" |

{m,g}awk '{ __[$+_]++ } END { for(_ in __) { print "",+__[_],_ } }' FS='^$' OFS='\t' | 

gsort -t$'\t' -k 3,3 -V

Only the post GROUP-BY summary rows are being sent to the sorting utility - which is far less system intensive sort compared to pre-sorting the input rows for no reason.

For small inputs, e.g. fewer than 1000 rows or so, just directly sort|uniq -c it.

    3   10.0.10.1
    1   10.0.10.2
    1   10.0.10.3

Sort may be omitted if order is not significant

uniq -c <source_file>

or

echo "$list" | uniq -c

if the source list is a variable