Deleting a dynamically created bootstrap form group with a Jquery onclick event

I have dynamically created a form-group with an input field and a delete button next to it, this button needs to remove: the button, the div it is contained in and the div that that div is contained in. So essentially the form-group which contains that button. Here is the PHP code:

                            <div class="form-horizontal" id="HighlightForm">
                                <div class="form-group">
                                    <label class="col-sm-4 control-label">
                                        Highlight(s)
                                    </label>
                                    <div id="highlightInput" class="input-group col-sm-8">
                                        <input type="input" name="aHighlight[]" class="form-control" />
                                        <span class="input-group-btn">
                                            <button id="add-highlight" class="form-control add-highlight"><i class="fa fa-plus"></i></button>
                                        </span>
                                    </div>
                                </div>
                            </div>

The jQuery portion:

$('#add-highlight').on('click', function ()
{
    $('<div class="form-group"><label class="col-sm-4 control-label"></label><div id="highlightInput" class="input-group col-sm-8"> <input type="input" name="aHighlight[]" class="form-control" /><span class="input-group-btn"> <button id="remove-highlight" class="form-control"><i class="fa fa-times"></i></button> </span></div></div>').appendTo('#HighlightForm');
});

$(document).on('click','#remove-highlight', function (e)
{
    e.preventDefault();
    $(this).parent('div').parent('div').remove();
});

I tried to address the button with $(this) and address the parent div and its parent div to remove the entire form group but that did not work.