Java Logger Console Stream Duplicate Output

I hope this question has a simple answer. I am trying to add a logger to my Java application. I was able to format the log file perfectly, but I ran into an issue when trying to add a ConsoleHandler to the logger to format the console output.

Once I added the ConsoleHandler, every log message is not printed out three times, twice with the proper formatting I want, and then once with the original format that I do not want.

Ill post my code below, not sure at all what I am doing wrong.
Any help will be greatly appreciated.

// remove and handlers that will be replaced
Handler[] handlers = logger.getHandlers();
for(Handler handler : handlers)
{
    if(handler.getClass() == ConsoleHandler.class || handler.getClass() == FileHandler.class)
        logger.removeHandler(handler);
}

// file handler
FileHandler fh = new FileHandler(file.toString());
fh.setFormatter(new MS2Formatter());
logger.addHandler(fh);

// console handler
ConsoleHandler ch = new ConsoleHandler();
ch.setFormatter(new MS2Formatter());
logger.addHandler(ch);

logger.setLevel(Level.INFO);

EDIT: Answer

Just wanted to post my final code here to help anyone with a similar problem.

logger = Logger.getLogger("My Logger");     
logger.setUseParentHandlers(false);

// remove and handlers that will be replaced
Handler[] handlers = logger.getHandlers();
for(Handler handler : handlers)
{
        if(handler.getClass() == ConsoleHandler.class)
            logger.removeHandler(handler);
}

// setup the file
File file = new File(location + "/" + fileName);

// file handler
FileHandler fh = new FileHandler(file.toString(), true);
fh.setFormatter(new MS2Formatter());
logger.addHandler(fh);

// console handler
ConsoleHandler ch = new ConsoleHandler();
ch.setFormatter(new MS2Formatter());
logger.addHandler(ch);

// remove and handlers that will be replaced
logger.setLevel(Level.INFO);