Set Windows process (or user) memory limit

Use Windows Job Objects. Jobs are like process groups and can limit memory usage and process priority.

Depending on your applications, it might be easier to limit the memory the language interpreter uses. For example with Java you can set the amount of RAM the JVM will be allocated.

Otherwise it is possible to set it once for each process with the windows API

SetProcessWorkingSetSize Function

No way to do this that I know of, although I'm very curious to read if anyone has a good answer. I have been thinking about adding something like this to one of the apps my company builds, but have found no good way to do it.

The one thing I can think of (although not directly on point) is that I believe you can limit the total memory usage for a COM+ application in Windows. It would require the app to be written to run in COM+, of course, but it's the closest way I know of.

The working set stuff is good (Job Objects also control working sets), but that's not total memory usage, only real memory usage (paged in) at any one time. It may work for what you want, but afaik it doesn't limit total allocated memory.

Per process limits

From an end-user perspective, there are some helpful answers (and comments) at the superuser question “Is it possible to limit the memory usage of a particular process on Windows”, including discussions of how to set recursive quota limits on any or all of:

• CPU assignment (quantity, affinity, NUMA groups),
• CPU usage,
• RAM usage (both ‘committed’ and ‘working set’), and
• network usage,

… mostly via the built-in Windows ‘Job Objects’ system (as mentioned in @Adam Mitz’s answer and @Stephen Martin’s comment above), using:

• the registry (for persistence, when desired) or
• free tools, such as the open-source Process Governor.

(Note: nested Job Objects ~may~ not have been available under all earlier versions of Windows, but the un-nested version appears to date back to Windows XP)

Per-user limits

As far as overall per-user quotas:

• ??
• It is possible that each user session is automatically assigned to a job group itself; if true, per-user limits should be able to be applied to that job group. Update: nope; Job Objects can only be nested at the time they are created or associated with a specific process, and in some cases a child Job Object is allowed to ‘break free’ from its parent and become independent, so they can’t facilitate ‘per-user’ resource limits.

(NTFS does support per-user file system ~storage~ quotas, though)

Per-system limits

Besides simple BIOS or ‘energy profile’ restrictions:

• VM hypervisor or Kubernetes-style container resource limit controls may be the most straightforward (in terms of end-user understandability, at least) option.

Footnotes, regarding per-process and other resource quotas / QoS for non-Windows systems:

• ‘Classic’ Mac OS (including ‘classic’ applications running on 2000s-era versions of Mac OS X): per-application memory limits can be easily set within the ‘Memory’ section of the Finder ‘Get Info’ window for the target program; as a system using a cooperative multitasking concurrency model, per-process CPU limits were impossible.
• BSD: ? (probably has some overlap with linux and non-proprietary macOS methods?)
• macOS (aka ‘Mac OS X’): no user-facing interface; system support includes, depending on version, the ‘Multiprocessing Services API’, Grand Central Dispatch, POSIX threads / pthread, ‘operation objects’, and possibly others.
• Linux: ‘Resource Manager’/limits.conf, control groups/‘cgroups’, process priority/‘niceness’/renice, others?
• IBM z/OS and other mainframe-style systems: resource controls / allocation was built-in from nearly the beginning