Stack different column values into one column in a pandas dataframe

you can use stack and pivot_table:

k = df.set_index(['ID', 'Date']).stack().reset_index()
df = k.pivot_table(index = ['ID',0], columns = 'level_2', values = 'Date', aggfunc = ''.join, fill_value= '').reset_index()
df.columns = ['ID', 'prior-current', 'start-date', 'end-date']

OUTPUT:

    ID prior-current  start-date    end-date
0    1             a                1/1/2019
1    1            a1    1/1/2019            
2    2             b                5/1/2019
3    2             c    5/1/2019   10/2/2019
4    2            c1   10/2/2019            
5    3             d               15/3/2019
6    3             e   15/3/2019    6/5/2019
7    3             f    6/5/2019    7/9/2019
8    3            f1    7/9/2019            
9    4             g              16/11/2019
10   4            g1  16/11/2019            

Explaination:

After stack / reset_index df will look like this:

   ID        Date  level_2   0
0    1    1/1/2019    Prior   a
1    1    1/1/2019  Current  a1
2    2    5/1/2019    Prior   b
3    2    5/1/2019  Current   c
4    2   10/2/2019    Prior   c
5    2   10/2/2019  Current  c1
6    3   15/3/2019    Prior   d
7    3   15/3/2019  Current   e
8    3    6/5/2019    Prior   e
9    3    6/5/2019  Current   f
10   3    7/9/2019    Prior   f
11   3    7/9/2019  Current  f1
12   4  16/11/2019    Prior   g
13   4  16/11/2019  Current  g1

Now, we can use ID and column 0 as index / level_2 as column / Date column as value.

Finally, we need to rename the columns to get the desired result.

My approach is to build and attain the target df step by step. The first step is an extension of your code using melt() and merge(). The merge is done based on the columns 'Current' and 'Prior' to get the start and end date.

df = pd.DataFrame({
    'ID': [1, 2, 2, 3, 3, 3, 4],
    'Prior': ['a', 'b', 'c', 'd', 'e', 'f', 'g'],
    'Current': ['a1', 'c', 'c1', 'e', 'f', 'f1', 'g1'],
    'Date': ['1/1/2019', '5/1/2019', '10/2/2019', '15/3/2019', '6/5/2019',
             '7/9/2019', '16/11/2019']
})
df2 = pd.melt(df, id_vars='ID', value_vars=['Prior', 'Current'], value_name='Prior_Current').drop('variable',1).drop_duplicates().sort_values('ID')
df2 = df2.merge(df[['Current', 'Date']], how='left', left_on='Prior_Current', right_on='Current').drop('Current',1)
df2 = df2.merge(df[['Prior', 'Date']], how='left', left_on='Prior_Current', right_on='Prior').drop('Prior',1)
df2 = df2.fillna('').reset_index(drop=True)
df2.columns = ['ID', 'Prior_Current', 'Start_Date', 'End_Date']

Alternative way is to define a custom function to get date, then use lambda function:

def get_date(x, col):
    try:
        return df['Date'][df[col]==x].values[0]
    except:
        return ''

df2 = pd.melt(df, id_vars='ID', value_vars=['Prior', 'Current'], value_name='Prior_Current').drop('variable',1).drop_duplicates().sort_values('ID').reset_index(drop=True)
df2['Start_Date'] = df2['Prior_Current'].apply(lambda x: get_date(x, 'Current'))
df2['End_Date'] = df2['Prior_Current'].apply(lambda x: get_date(x, 'Prior'))

Output