Given that there are not that many 1-byte characters, but only 256, they can quickly be precomputed at your script startup:
declare -ag CHARS=()
for REPLY in {{0..9},{a..f}}{{0..9},{a..f}}; do
printf -v CHARS[${#CHARS[@]}] "\x$REPLY"
done
(I reused the discardable REPLY variable, but you can local your own inside an init function)
Then...
$ echo "${CHARS[65]}"
A
$ i=65
$ echo "${CHARS[i]}"
A
Beware of \x00, whose value is not properly handled in Bash because it is the string terminator character:
$ var=$'qwe\x00rty'
$ echo ${#var}
3
$ echo "<${var}>"
<qwe>
So, "${CHAR[0]}" will always be a problem under Bash.
This way, you avoid output capture, inline text substitution and re-parsing once and again for every time you need to process one character; which is even worse for subprocess IPC through xxd, awk or perl.