If I do not specify any --gres=gpu:1 option then the process will use up all GPUs in the compute node.
We only use Slurm for GPU sharing so we would like that every process be assigned one GPU automatically... Is it possible to specify that by default srun --gres=gpu:1?
You can set a default for --gres by setting the SBATCH_GRES env variable to all users, for instance in /etc/profile.d on the login node. Simply create a file in there, that has the following content:
export SBATCH_GRES=gpu:1
Note that the documentation says
Note that environment variables will override any options set in a batch script
so people who will want to use more than one, or not use a GPU at all will need to override this default using the command line option, and won't be able to override it with a #SBATCH --gres line in their submission script.
Another option would be, to set the CUDA_VISIBLE_DEVICES to an empty string for all users by default. Then, in jobs that request GPUs, the variable will be modified by Slurm according to the request, and jobs that do not make the GPU request will not 'see' the GPUs.
If users are likely to play the system (the CUDA_VISIBLE_DEVICES variable can be overwritten by the users), then you will have to set cgroups.