Prevent container from growing when legend text gets displayed

I'm trying to stop the container of my image and subtext from growing when the text becomes displayed on hover. At the moment on hover you can see the portfolio__item element resize to fit the text in and it's background color appear at the bottom.

I've tried using a position: absolute on the text but it then makes positioning the text in the middle of the image almost impossible.

Here's a code pen i've made that's a replica of what i'm working on: https://codepen.io/Olivier42M/pen/oNzwGEL

Html:

<div class="wrapper"> 
 <div class="portfolio">
    <a href="" class="portfolio__item">
      <img src="https://i.imgur.com/MKuGA17.jpeg" alt="" class="portfolio__img">
      <div class="portfolio__text">Legend text</div>
    </a>
     <a href="" class="portfolio__item">
      <img src="https://i.imgur.com/MKuGA17.jpeg" alt="" class="portfolio__img">
      <div class="portfolio__text">Legend text</div>
    </a>
     <a href="" class="portfolio__item">
      <img src="https://i.imgur.com/MKuGA17.jpeg" alt="" class="portfolio__img">
      <div class="portfolio__text">Legend text</div>
    </a>
     <a href="" class="portfolio__item">
      <img src="https://i.imgur.com/MKuGA17.jpeg" alt="" class="portfolio__img">
      <div class="portfolio__text">Legend text</div>
    </a>
     <a href="" class="portfolio__item">
      <img src="https://i.imgur.com/MKuGA17.jpeg" alt="" class="portfolio__img">
      <div class="portfolio__text">Legend text</div>
    </a>
   
 </div>
</div>

Css:

body{
  margin:0;
  padding:0;
}

.wrapper{
  text-aligne: center;
}

.portfolio {
    display: grid;
    grid-template-columns: repeat(auto-fit, minmax(300px, 1fr));
}

.portfolio__item {
    background-color: orange;
    text-decoration: none;
    overflow: hidden;
    
}

.portfolio__img {
    transition: 
        transform 750ms cubic-bezier(.5, 0, .5, 1),
        opacity 250ms linear;
}


.portfolio__item:focus {
    position: relative;
    z-index: 2;
}

.portfolio__img:hover,
.portfolio__item:focus .portfolio__img{
    transform: scale(1.2);
    opacity: .75;
}

.portfolio__text{
    z-index: 3;
    position: relative;
    color: white;
    margin-left: auto;
    font-size: 3em;
    bottom: 50%;
    opacity: 0;
    opacity: 0;
    height: 0;
    transition: 
        opacity 500ms linear;
    overflow: hidden;
}

.portfolio__item:hover .portfolio__text{
    opacity: 1;
    height: auto;
}