I am a bit late but I hope I can clear some things up.
Encoder:
Chooses what part of your Hardware transcodes the video, whereas software stands for CPU only and hardware is either your graphics card or the integrated graphics of your CPU (Quicksync for Intel processors). The two latter are generally much faster as they are built for heavily parallelized workloads, using simple mathematical algorithms to transcode a video very fast at the cost of image quality or higher bitrate, usually both. A CPU can use advanced mathematical algorithms, which dramatically increases the time spend for transcoding, depending on what settings you use (see below). However, it has by far the best image quality at a given bitrate or a far lower bitrate at the same quality.
Encoding options:
H264/avc: Older codec, faster but worse quality than
H265/HEVC: As the name says, the high efficiency video codec saves bitrate at the same image quality, but increases transcoding time, especially for the CPU.
Both software and hardware encoding can make use of both codecs. Generally speaking, if you want the lowest bitrate possible, use software h265, if you dont care about file size but want good image quality, choose hardware h265. Hardware H265 is available on all modern graphics cards starting from AMDs r9 300 series and NVIDIA GTX 900 series or newer aswell as 6th generation or newer Intel processors with iGPU (Quicksync).
For CPU encoding: If you go to the advanced encoder tab you can add codec options, eg. presets to control the speed and quality. Here is a list of x265 presets, I do not recommend choosing any faster preset than veryfast as it decreases the blocksize form 64x64 to 32x32, at this point you could just use hardware encoding, there wouldn't be that much of a difference except transcoding speed ofc. Example attached.
Worth to note is that the CRF slider in the basic tab determines a constant image quality displayed in a logarithmic number (20 by default), therefore the lower the number = the better the quality. The bitrate however stays variable and can spike up really fast without any improvement in image quality if you use an overkill number.
Playback settings:
Higher performance uses a faster preset, which speeds up transcoding time at the cost of image quality. Higher quality, as the name tells you, increases the quality but requires better hardware, mostly better cpu.