Do iterators define semantics for what `next()` should return after an error?

user_272520 0
Viewed 80

The basic question is: if an implementation of the Iterator trait returns a Result<T, E>, what should the iterator do after an error is returned from next() that makes it impossible to continue iterating?

Some context:

As a learning project, I'm attempting to implement a parsing library to parse STUN messages, as defined in RFC5389 in Rust. STUN is a binary network protocol, and as such I would be parsing byte slices. Specifically, the protocol specifies that zero or more dynamically-sized attributes can be encoded into the message. Thus, I am trying to construct an iterator which can be used to iterate over the bytes, yielding subslices for each attribute.

Thus, I have something like this...

pub struct StunAttributeIterator<'a> {
    data: &'a [u8],
}

impl<'a> Iterator for StunAttributeIterator<'a> {
    type Item = Result<StunAttribute<'a>, StunAttributeError>;

    fn next(&mut self) -> Option<Self::Item> {
        if self.data.len() == 0 {
            return None;
        }

        // Ensure there are enough bytes left in the stream to parse the next attribute.
        if self.data.len() < ATTRIBUTE_TYPE_LENGTH_BYTES {
            return Some(Err(StunAttributeError::UnexpectedEndOfStream));
        }

        // Parse the data and get a slice of the dynamic bytes to return
        let data = ...;
        // Modify the iterator to have the slice move to the start of the next attribute
        self.data = ...;
        return Some(Ok(StunAttribute { data }));
    }
}

There are a number of things that could go wrong here, and I've included an example in the if statement. If something goes wrong, it's a good indication that the byte stream being parsed is malformed, and thus there is no reason to continue attempting to parse.

On the one hand, I could just leave the code as-is, but I worry that this could create some infinite loops; if the error is ignored next() could be continuously called returning an Err each time. On the other, I could change the iterator so that on a subsequent call to next() after the error None is returned.

Are there guidelines/best practices for what to do as the implementer of the iterator this situation? I know some iterator adapters are aware of iterators that return Result<T, E>, but probably not all.

rust error-handling iterator
1 Answers

Let's look at what the docs say:

Returns None when iteration is finished. Individual iterator implementations may choose to resume iteration, and so calling next() again may or may not eventually start returning Some(Item) again at some point.

So as far as the Iterator contract is concerned, you can return anything from next() at any time -- even returning Some after previously returning None.

(The FusedIterator tag indicates that the iterator's next() promises to only return None after it has returned None before.)

All that to say, there isn't a required behavior. You're not even talking about None here, you're talking about Some(Err(_)), so even if the contract of FusedIterator wasn't a separate thing and was instead mandated by Iterator, you'd still technically be fine here no matter what you choose to do.

There are a few Iterator utilities that can interact specifically with a sequence of Results. For example, you can .collect() an iterator of Result<T, E> into a Result<Vec<T>, E>, but this assumes that you only want to look at the produced sequence if no errors occurred whatsoever.

I would argue that there are a few sensible things you could do after yielding an error:

  1. If the failure is permanent, yield None afterwards. The following next() call could restart the operation, if you would like. (Or, return None forever, and then it would be a good idea to implement FusedIterator to communicate this.)
  2. If the failure is temporary (the operation can be retried) then you could try it again, and leave it up to the caller to figure out when they want to stop.
  3. Either of the prior options depending on either configuration or a method call. For example, you could have a retries argument to your iterator's constructor/factory indicating how many sequential failures to return before giving up and returning None, or you could add a fn retry() to your type. If next() returns an error, it would return None on the following call unless retry() was called immediately prior, in which case it would retry the operation.

Whatever, you decide, document it clearly. The second case has the potential to result in a never-ending sequence of errors, which makes not retrying the operation the safest thing to do.

Having said that, I have written a similar type of iterator and I had it retry in the case of an error. Some of my callers use ? on each element to bail on an error, others implement retry logic that will eventually bail if too many errors happen.

user_5012500
Related