How to compare keys of vector in a HashMap?

You want a max_by comparison function which compares the first element to find which is greatest and the second to find which is smallest. Let's deal with the first criterion first: the first element should be largest.

a.0[0].cmp(&b.0[0])

a.0 and b.0, as you've likely already surmised, select the key from the key-value pair provided. Then we use ordinary [] indexing to get the first element of the vector.

Now, I'm assuming for the purposes of the question that we want lexicographic ordering, also called dictionary order. That is, you've listed two criteria, and I'm assuming the first takes precedent, so that for instance [10, 2] should be favorable to [9, 1], since the first element is bigger, despite the fact that the second is not smaller.

With that assumption in mind, we can expand our comparison. To reverse an ordering (i.e. to select a smallest rather than largest element) we can simply switch the order of arguments.

b.0[1].cmp(&a.0[1])

Then we only want to use this comparison if the first failed us, i.e. if the first was equal.

match a.0[0].cmp(&b.0[0]) {
  Ordering::Equal => b.0[1].cmp(&a.0[1]),
  x => x
}

If we put this in your max_by function, we get [10, 1], as desired.

But we can actually do one better. See, lexicographic ordering is fairly common, so common that it's built-in to Rust. We can use the method Ordering::then as follows.

a.0[0].cmp(&b.0[0]).then(b.0[1].cmp(&a.0[1]))

And this works identically to the previous example.

Complete example:

let mut queue : HashMap<Vec<u8>, Vec<u8>> = HashMap::new();

queue.insert(vec![5 as u8, queue.keys().len() as u8], vec![0]);
queue.insert(vec![10 as u8, queue.keys().len() as u8], vec![1]);
queue.insert(vec![10 as u8, queue.keys().len() as u8], vec![2]);
queue.insert(vec![3 as u8, queue.keys().len() as u8], vec![2]);
queue.insert(vec![4 as u8, queue.keys().len() as u8], vec![3]);
queue.insert(vec![6 as u8, queue.keys().len() as u8], vec![4]);

let key = queue
  .iter()
  .max_by(|a, b| {
    a.0[0].cmp(&b.0[0]).then(b.0[1].cmp(&a.0[1]))
  })
  .map(|(k, _v)| k);

println!("{:?}", key);

HashMap is the wrong data structure for this task. In order to solve this problem, you have to access every element of the map. If you have control over your data, you should store it in a BTreeMap instead. Then the solution to your problem becomes:

fn get_first_of_max(queue: &BTreeMap<Vec<u8>, Vec<u8>>) -> Option<Vec<u8>> {
    let max_key_prefix = queue.keys().rev().next()?;
    queue.range((
        Bound::Included(vec![max_key_prefix[0], 0]), 
        Bound::Unbounded,
    )).next().map(|x| x.0.to_owned())
}

Given what you're doing with your values, I'd also prefer a (u8, usize) for the key. Combined with the above, you get:

use std::collections::BTreeMap;
use std::ops::Bound;

fn get_first_of_max(queue: &BTreeMap<(u8, usize), Vec<u8>>) -> Option<(u8, usize)> {
    let max_key_prefix = queue.keys().rev().next()?;
    queue.range((
        Bound::Included((max_key_prefix.0, 0)), 
        Bound::Unbounded,
    )).next().map(|x| x.0.to_owned())
}

fn main() {
    let mut queue = BTreeMap::new();

    queue.insert((5, queue.keys().len()), vec![0]);
    queue.insert((10, queue.keys().len()), vec![1]);
    queue.insert((10, queue.keys().len()), vec![2]);
    queue.insert((3, queue.keys().len()), vec![3]);
    queue.insert((4, queue.keys().len()), vec![4]);
    queue.insert((6, queue.keys().len()), vec![5]);

    println!("{:?}", get_first_of_max(&queue));
}