How to group this data to be filterable

user_10725504 0
Viewed 102

I am using dc.js to create cross-filterable charts with an array of "changesets" that have a schema like the following:

  id,
  first_name, last_name, user_id,
  created_at,
  num_changes,
  hashtags: [str],
  total_add, total_mod, total_del,
  buildings_add, buildings_mod, buildings_del,
  pois_add, pois_mod, pois_del,
  roads_add, roads_mod, roads_del,
  road_km_add, road_km_mod, road_km_del,
  waterways_add, waterways_mod, waterways_del,
  waterway_km_add, waterway_km_mod, waterway_km_del
}

Question

I seek to create filterable stacked bar charts with the bars representing add/mod/del and stacks representing the types of data changed buildings/pois/roads/waterways/road_km/waterway_km.

any 1 changeset can have any combination of these fields and so you cannot pair each changeset with 1 modification type. Is there some better way to accomplish this grouping where I can apply filters to the chart?

code I've tried

I have the chart display working with the correct data, but the way I've set it up has made it so the chart cannot be filtered. Initially I had written the dimension as crossfilter.dimension(d => ['add', 'mod', 'del'], true) so that each changeset shows up in each bin. but because all changesets would share add/mod/del, nothing is filtered.

I then saw the filter stacks example here: https://github.com/dc-js/dc.js/blob/develop/web-src/examples/filter-stacks.html

And I tried to run a multi-key dimension for the edits

let editDim = ndx.dimension(d => {
  let rt = []
  stackKeys.forEach(key => {
    editStacks.forEach(stack => {
      if (d[sAcc(stack,key)]) {
        rt.push(key + '.' + stack)
      }
    })
  })
  return rt
}, true)

this method looks so close but filtering the stacks do not produce the correct results on other charts. It seems like no matter what I choose to filter out of this chart, the other charts produce 0.

Here is a jsfiddle where I have 1 stacked bar chart using the multi-key method, 1 stacked bar chart using the ['add', 'mod', 'del'] key method and 1 regular bar chart to compare results/filtering with.

To separate the changesets into groups, I have used a custom reducer which transforms the data to something that looks like

{
    key: 'add',
    value: {
        add: {
            buildings': 42,
            pois: 12,
            roads: 1,
            waterway: 2,
            waterway_km: 0.003,
            road_km: 0
        },
        mod: {...}
        del: {...}
    }
}

The dimension is grouped into ['add', 'mod', 'del'] and the stacks are created using

const editStacks = ['buildings', 'pois', 'roads', 'waterways']
editStacks.forEach((stack, i) => {
    // first is group, others are stacked
    let action = i ? 'stack' : 'group'
    chart[action](group, stack, d => d.value[d.key][stack])
})

in the multi-key method, the values are transformed to be just {key: 'add', value: {building, roads, pois, waterways}} using this function

all: function () {
  var all = group.all()
  var m = {}
  all.forEach(kv => {
    let [k,s] = kv.key.split('.')
    m[k] = m[k] || {}
    m[k][s] = kv.value[k][s]
  })
  return Object.keys(m).map(key => {
    return {key, value: m[key]}
  })
}
crossfilter dc.js data-manipulation javascript
1 Answers

So to get the stacked bars filterable by each section I had to use a combination of the answers found in dc.js - Creating a row chart from multiple columns and enabling filtering and in Plotting aggregated data with sub-columns in dc.js

to construct a filterable dimension you must use the methods described in the first link to create a custom filterHandler on your dimension which is just grouped up by element

let dimension = ndx.dimension(d => d)

then I constructed my group by using groupAll and creating an object of { key.stack: value }} using the reduce functions. Then I converted it back to a standard group by creating an "all" method for the group as described in the second link.

function reduceAdd (p, v) {
    keys.forEach(k => {
        stacks.forEach(s => {
            p[`${k}.${s}`] += v[accessor(k, s)] || 0
        })
    })
    return p
}
function reduceRemove (p, v) {
    keys.forEach(k => {
        stacks.forEach(s => {
            p[`${k}.${s}`] -= v[accessor(k, s)] || 0
        })
    })
    return p
}
function reduceInit () {
    let p = {}
    keys.forEach(k => {
        stacks.forEach(s => {
            p[`${k}.${s}`] = 0
        })
    })
    return p
}

function stackedGroup (group) {
    return {
        all: function () {
            var all = Object.entries(group.value()).map(([key, value]) => ({ key, value }))
            var m = {}
            all.forEach(kv => {
                let [k, s] = kv.key.split('.')
                m[k] = m[k] || {}
                m[k][s] = kv.value
            })
            return Object.keys(m).map(key => {
                return { key, value: m[key] }
            })
        }
    }
}

let group = dimension.groupAll().reduce(reduceAdd, reduceRemove, reduceInit)
group = stackedGroup(group)

finally you must redefine the filter handler as mentioned in the first link. This filter is what I use to filter by "changesets that have at least 1 x" where "x" is some type of edit such as "add buildings" or "modified road"

chart.filterHandler((dim, filters) => {
    if (filters && filters.length) {
        dim.filterFunction((r) => {
            return filters.some((c) => {
                //the changeset must have a value in this field to be left in the chart
                let [stack, field] = c[0].split('.')
                return r[accessor(field, stack)] > 0
            })
        })
    } else {
        dim.filter(null)
    }
    return filters
})

After that, the hashtags chart was showing unexpected results, so I had to do a similar dimension & grouping for the hashtags chart as well. I believe this is because the hashtags chart was defined with an array dimension, but grouping it as d => d and defining a custom filterHandler has produced results I am pleased with. Both charts are filterable to find results to questions such as "how many changesets with the hashtag #Kaart have road modifications".

Thanks again @Gordon!

user_107255040
Related