is it possible to count specific hex number in a file using fgetc?

You have a large number of small errors. They can be summarized as follows:

• while (!feof(in)) reads one-character-too-many. Look at the logic of the loop. After reading the last character, you check !feof(in) (which hasn't occurred yet) and then call b = fgetc (in); again (which now returns EOF), and then you blindly assign a[i] = b;. That's Why is while ( !feof (file) ) always wrong? Simply control your read-loop with the return from your read-function.
• You use the wrong type with scanf(). %x requires an unsigned int* value, but you pass type int*. This will result in problems with signed and unsigned type mismatch. This is readily apparent when you compile with warnings enabled.
• You fail to validate whether your open of in and out succeed. Always validate every file open operation.
• You fail the validate the return of scanf(). You cannot use scanf() correctly unless you validate the number returned is equal to the number of valid conversions expected.
• Since you write to b.txt you should validate the fclose(out). Always validate your close-after-write to ensure you catch any write error that occurs after the last value written by your code.
• There is no need to loop over all 1000 elements of your array. You know the number of elements filled from the value of i. Just loop over the elements filled using a separate loop variable (j shown below).
• Lastly, when you need input from your user, don't leave the user staring at a blinking cursor on the screen wondering whether the program is hung or what is going on, prompt the user for input.

Putting all the pieces together, you could do something similar to the following:

#include <stdio.h>
#include <stdlib.h>

#define MAXC 1000   /* if you need a constant, #define one (or more) */

int main (void)
{
    FILE *in, *out;
    unsigned char a[MAXC] = { 0 };
    int b;
    unsigned u;                   /* unsigned value required for scanf() */
    int count = 0, i = 0, j;
    
    in = fopen ("a.dat", "rb");
    out = fopen ("b.txt", "wb");
    
    if (!in) {  /* always validate every file open */
        perror ("fopen-a.dat");
        return 1;
    }
    
    if (!out) {  /* always validate every file open */
        perror ("fopen-b.txt");
        return 1;
    }
    
    /* protect array bound - use read function to control loop */
    while (i < MAXC && (b = fgetc (in)) != EOF) {
        a[i] = b;
        i++;
    }
    
    fputs ("enter 8-bit hex value to find: ", stdout);
    if (scanf ("%x", &u) != 1) {      /* validte every user-input */
        fputs ("error: invalid hex input.\n", stderr);
        return 1;
    }
    
    for (j = 0; j < i; j++) {     /* iterate over values read from file */
        if (a[j] == u) {
            count++;
        }
    }

    fprintf (out, "%d\n", count);
    printf ("%d\n", count);
    
    if (fclose (out) == EOF) {    /* always validate close-after-write */
        perror ("fclose-out");
    }
    fclose (in);
}

Example Use/Output

Compiling your code with full warnings enabled you can do:

$ gcc -Wall -Wextra -pedantic -Wshadow -std=c11 -O3 -o bin/readwriteucbin readwriteucbin.c

Running your code on the binary input provided you would get, e.g.

$ ./bin/readwriteucbin
enter 8-bit hex value to find: 0xff
1

Or where more than one value is matched, e.g.

$ ./bin/readwriteucbin
enter 8-bit hex value to find: 1
2