Behind the scenes at Civis, we take protecting client data and privacy very seriously. We invest substantial resources and Security/DevOps engineering time in technology and processes to protect our systems and client data. In addition to our own engineers, we rely on a variety of security technologies, including:
Amazon Web Services (AWS) and their offerings, such as Amazon Inspector
Next-generation endpoint protection for laptops and servers, such as SentinelOne
Laptop and phone management software that enforces security policies
A cloud-based log analytics tool in the form of Sumo Logic
Being an analytics company, we have a soft spot for analytics products such as Sumo Logic. We use it to collect and securely transport security and application logs from networking devices, applications, laptops, OSSEC agents, and AWS EC2 instances to the Sumo Logic application. We then use Sumo Logic to search for issues and attacks, as well as provide necessary audit controls as we pursue HIPAA and SOC2 Type II compliance and certification.
One such audit control is detecting when engineers login and logout of our AWS EC2 instances. Civis utilizes a group account called “civis-ec2” and unique SSH public/private key pairs for these Civis staff members in the civis-ec2 user’s authorized_keys file. This is done for simplicity of administration in DevOps playbooks and deployed EC2 instances, versus having to manage local accounts for every user on every instance.
However, the OSSEC agent login/logout events (below) sent to Sumo Logic do not by themselves contain the necessary user details to map the civis-ec2 user to the unique Civis staff member who created the SSH session. We do have two pieces of data to work with to determine who the user was and meet HIPAA and SOC2 compliance requirements: the user’s SSH public key fingerprint in the login event and the sshd process ID (PID) in both events (all in bold text below).
EC2 SSH Login Event from OSSEC:
Src IP: 10.10.10.80
May 17 15:26:00 a3 sshd: Accepted publickey for civis-ec2 from 10.10.10.80 port 51836 ssh2: RSA 22:f6:fc:22:40:10:e2:af:1d:19:3d:59:88:74:4a:1b
EC2 SSH Logout Event from OSSEC:
May 17 15:26:02 a3 sshd: pam_unix(sshd:session): session closed for user civis-ec2
To correlate the unique Civis user to the user that logged in as “civis-ec2”, we’ll use the list of all SSH public keys in our EC2 servers’ authorized_keys file. This file contains the full public SSH key for each user and their associated username in the SSH public key (e.g. jhollandcivis).
Using this data, we wrote a script to generate a nested if-then-else statement to use in a Sumo Logic query. The script takes each user’s public key and generates the MD5 formatted fingerprint along with their associated username, and then creates the nested if-then-else format with the proper number of closing parenthesis (as shown below). The else condition “SSH Key Not Mapped” is then applied to the key_name variable in the event a new key was added to the authorized_keys file but is not mapped to a user in our sumo query.
Having a script that generates the nested if statement is useful as we add more staff and rotate keys.
We then use this statement within a larger query we’ll run against our OSSEC logs in Sumo Logic:
| formatDate(_messageTime, “MM/dd/yyyy HH:mm:ss:SSS”) as messageDate
| parse regex “\((?<target_hostname>.*?)\)” nodrop
| parse regex “\) (?<target_ip>\d+\.\d+\.\d+\.\d+)->” nodrop
| parse regex “Src IP: (?<src_ip>\d+\.\d+\.\d+\.\d+)” nodrop
| parse regex “ssh2: RSA (?<rsa_fingerprint>.*)$” nodrop
| parse regex ” (?<temp>Accepted publickey for civis-ec2) ” nodrop
| replace (temp,”Accepted publickey for civis-ec2″,”User Login via SSH”) as login_event
| parse regex ” (?<temp2>session closed for user .*?)$” nodrop
| replace (temp2,”session closed for user civis-ec2″,”User logged out via SSH”) as logout_event
| parse regex ” sshd\[(?<ssh_pid>\d+)\]: ”
| if(rsa_fingerprint=”4c:8d:a1:af:49:9c:06:bf:95:2c:a7:7a:84:66:bb:7e”,”john-civis”,if(rsa_fingerprint=”21:f1:fc:23:43:20:b2:bf:1d:19:3f:41:28:74:1c:1b”,”jhollandcivis”,if(rsa_fingerprint=”1b:06:1e:6e:9f:fa:1f:03:9b:ba:21:99:d0:fc:81:8e”,”jane-civis”,if(rsa_fingerprint=”d9:b1:aa:71:58:44:a2:38:f2:27:ff:06:1c:8e:55:79″,”civis-pete”,”SSH Key Not Mapped”)))) as key_name
(parse “sshd[*]: Accepted publickey” as A) as login,
(parse “sshd[*]: pam_unix” as B) as logout
on login.A = logout.B
| formatDate (fromMillis(login__messageTime), “MM/dd/yyyy HH:mm:ss:SSS”) as login_messageTime
| formatDate (fromMillis(logout__messageTime), “MM/dd/yyyy HH:mm:ss:SSS”) as logout_messageTime
| logout__messageTime – login__messageTime as session_time
| (session_time / 60000) as session_duration_in_minutes
| count by login_messagedate,login_login_event,login_key_name,login_ssh_pid,login_src_ip,login_target_ip,login_target_hostname,logout_messagedate,logout_logout_event,logout_ssh_pid,session_duration_in_minutes
| sort by login_messagedate,login_ssh_pid,_count
Running this query in Sumo Logic, we get the following table that correlates SSH logins and subsequent logouts for the user “jhollandcivis”. Note that each event contains the login and logout times, the user who logged in, source and target IP addresses, target hostname, and the duration of the SSH session (in minutes). Also note the fact that the login and logout events both have the same SSH PID value, thus correlating the login event to the logout event.
Now that we have this data, we can answer questions such as:
- Who logged into a specific EC2 instance and when?
- How long were they logged in?
- When did they log out?
- What was the PID of their SSH session?
Additionally, we can see when a user logs into an EC2 instance with a new SSH public key our query does not know about (note the login_key_name value is “SSH Key Not Mapped”):
Finally, we’ll want to know when this happens, so we schedule a query to look for “SSH Key Not Mapped” events and run it every 15 minutes and have it email us if more than zero (0) results are returned:
When we receive the email with the query results, we can run our script again to generate a new nested if-then-else statement with the new user’s SSH key fingerprint and username, update our search query, and resolve “SSH Key Not Mapped” events.
Utilizing Sumo Logic, we now have the requisite audit information and auditing capability to react to security issues as well as satisfy our compliance requirements.