Is it possible to check if program is already running before trying to install it? (Inno Setup)

If it is your application, make it create a mutex. Then you can use AppMutex directive.

[Setup]
AppMutex=MyProgMutex

If you cannot modify the application, you need to code the check for running application in Inno Setup. You can for example use IsAppRunning function from the answer by @RRUZ to How to check with Inno Setup, if a process is running at a Windows 2008 R2 64bit? in InitializeSetup event function.

function InitializeSetup(): Boolean;
var
  Answer: Integer;
begin
  Result := True;
  while IsAppRunning('MyProg.exe') do
  begin
    Answer := MsgBox('Program is running, please close it', mbError, MB_OKCANCEL);
    if Answer = IDCANCEL then
    begin
      Result := False
      Exit;
    end;
  end;
end;

Based on a similar question on uninstaller:
Preparing to Uninstall like Preparing to Install Page - Inno Setup