In /etc/pam.d/system-auth:
auth required pam_env.so
auth required pam_tally.so onerr=fail deny=3 <-- Need tally to record failures
auth sufficient pam_unix.so nullok try_first_pass
auth requisite pam_succeed_if.so uid >= 500 quiet
auth required pam_deny.so
account required pam_tally.so reset <-- Same as above
account required pam_unix.so
account sufficient pam_succeed_if.so uid < 500 quiet
account required pam_permit.so
Then you need to run a couple commands to active everything.
Set lock out at 3 failed login attempts:
faillog -m 3
Exclude root from this lockout mechanism:
faillog -u root -m 0
To enforce failed login lockout after adding a user:
faillog -u {username} -m 3
If a user should fail to login 3 times, you must reset their tally before they will be able to login again:
faillog -u {username} -r



