Effects of switching between /SUBSYSTEM:CONSOLE to /SUBSYSTEM:WINDOWS in a DLL

Viewed 6278

I couldn't find anything useful on the MSDN for this case. Using Dependency Walker, in the module list, I see a mixed usage of Console and GUI.

Does this have an impact when compiling a DLL?

2 Answers

My below reply is just some findings.

I tried to create below types of project within VS2015:

  • Win32 console app project
  • Win32 DLL project
  • Win32 Windows app project

Below are their complete linking options:

Win32 console app project

/OUT:"C:\Temp\ConsoleApplication3\Debug\ConsoleApplication3.exe" /MANIFEST /NXCOMPAT /PDB:"C:\Temp\ConsoleApplication3\Debug\ConsoleApplication3.pdb" /DYNAMICBASE "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "comdlg32.lib" "advapi32.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "uuid.lib" "odbc32.lib" "odbccp32.lib" /DEBUG /MACHINE:X86 /INCREMENTAL /PGD:"C:\Temp\ConsoleApplication3\Debug\ConsoleApplication3.pgd" /SUBSYSTEM:CONSOLE /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /ManifestFile:"Debug\ConsoleApplication3.exe.intermediate.manifest" /ERRORREPORT:PROMPT /NOLOGO /TLBID:1

Win32 DLL project

/OUT:"C:\Temp\ConsoleApplication3\Debug\Win32DLLProject1.dll" /MANIFEST /NXCOMPAT /PDB:"C:\Temp\ConsoleApplication3\Debug\Win32DLLProject1.pdb" /DYNAMICBASE "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "comdlg32.lib" "advapi32.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "uuid.lib" "odbc32.lib" "odbccp32.lib" /IMPLIB:"C:\Temp\ConsoleApplication3\Debug\Win32DLLProject1.lib" /DEBUG /DLL /MACHINE:X86 /INCREMENTAL /PGD:"C:\Temp\ConsoleApplication3\Debug\Win32DLLProject1.pgd" /SUBSYSTEM:WINDOWS /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /ManifestFile:"Debug\Win32DLLProject1.dll.intermediate.manifest" /ERRORREPORT:PROMPT /NOLOGO /TLBID:1

Win32 Windows app project

/OUT:"C:\Temp\ConsoleApplication3\Debug\Win32WindowsProject1.exe" /MANIFEST /NXCOMPAT /PDB:"C:\Temp\ConsoleApplication3\Debug\Win32WindowsProject1.pdb" /DYNAMICBASE "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "comdlg32.lib" "advapi32.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "uuid.lib" "odbc32.lib" "odbccp32.lib" /DEBUG /MACHINE:X86 /INCREMENTAL /PGD:"C:\Temp\ConsoleApplication3\Debug\Win32WindowsProject1.pgd" /SUBSYSTEM:WINDOWS /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /ManifestFile:"Debug\Win32WindowsProject1.exe.intermediate.manifest" /ERRORREPORT:PROMPT /NOLOGO /TLBID:1

So we can see, for a /DLL, /SUBSYSTEM:WINDOWS is defined.

Then I tried to build a DLL with 3 different SUBSYSTEM values:

  • CONSOLE
  • WINDOWS
  • NOT SET

There are other values as well, but I can only build successfully with the above 3. Other values will lead to linking errors, which means some external symbols are needed.

The full list of possible SUBSYSTEM values is:

enter image description here

You can use the CFF Explorer to inspect the PE/COFF binaries.

The SUBSYSTEM header field locates at the file offset 0x14C of a PE/COFF file. It is part of the Optional Header.

/DLL /SUBSYESTEM:CONSOLE

enter image description here

/DLL /SUBSYESTEM:WINDOWS

enter image description here

/DLL and no /SUBSYESTEM option (by selecting NOT SET in VS2015 project properties page

enter image description here

Interestingly, the NOT SET and WINDOWS values lead to the same content in the binary header.

And I compared the whole binary of the 3 DLLs. It seems the rest of the binaries are all the same, except for some timestamp and debug info.

This is just some fact I found. How /SUBSYSTEM option affects the binary behavior depends on how the loader interprets this field.

In one of the projects I participated in, the /SUBSYSTEM:CONSOLE and /DLL are used together, which is different from the default combination for a DLL project. But it seems nothing bad happens. So I agree with @Frédéric Hamidi that this flag has no functional impact to a DLL.

Related