X264 vs NVENC vs VCE - Which encoder should you use?
The internet have various opinions regarding these three encoders and which is the ultimate winner compared to the others. Not all encoders are born equally despite being determined by the same factors. Let us have a look at them all and then you can try them out for yourself and decide what works for you.
Instead of going all technical about it, look at how it performs and what it can do for you with your build. Sure one of them outputs better quality, but that makes no difference if you can not handle that encoder.
What graphics card do you have?
The very first thing to look as is your choice of graphics cards manufacturer as NVENC is only available on NVidia cards - above the 600 series, after 2012 - and VCE on certain AMD cards. X264 is, however, available on almost all modern processors.
With that knowledge, you can put up the contest between NVENC and VCE, but that would be a very technological comparison. In the end, it would probably not make you change if you are loyal to a brand already. Therefore, let us make this about either x264 vs NVENC -or- VCE.
Performance and quality
If you ask the nerdy people then you should always go with x264 (CPU) encoding because it makes much more precise calculations compared to a GPU. This is, however, not too relevant when you are livestreaming as the data is a stream of content that does not need to be completely accurate as long as the coloring and overall visual is good.
What kind of processing do the games you are playing demand from your hardware? If you play a game like Counter-Strike, then it will be heavily CPU loaded compared to Battlefield V (5) that takes up a lot of your GPU.
Though, you do have an advantage through the GPU as the encoding is going through a Buffer Output chip. This basically means that all your graphics card renderings are directly readable by software like OBS and then create less demand. Just bear in mind that if your game is very demanding from your GPU that the buffer chip may not be able to follow through and then you need CPU-Encoding.
As this is all mathematically endused you should be aware that the lower bitrate you are using can create a bigger gap between the different encoders quality based on how precisely the encoding is. This does also depend on what encoding settings you are using.
Disclaimer: This article was published with intentions of letting you know what kind of encoder you should use for your livestream based on the technology and how well your hardware is able to perform. If you want precise benchmarks then please research this on your own as this can be very extensive information and hard to understand for non-technical users.