Welcome to LabVerge blog.
The Cybersecurity and Infrastructure Security Agency (CISA) is aware of active exploitation of SolarWinds Orion Platform software versions 2019.4 through 2020.2.1, released between March 2020 and June 2020.
It is recommended to do an incident analysis to get evidence of the attack. After conducting the research LabVerge made a shortlist of cases that can be broken down into several categories:
1. Network isolation
- Although some guides are saying that you need to update SolarWinds to the latest version LabVerge highly don’t recommend doing only it. If you confirmed that the backdoor version was in your servers, do a snapshot of the servers, including RAM ( if possible ), and after that shutdown them asap. We saw situations when SolarWinds servers were updated before making snapshots. It destroyed some evidence on servers.
- Don’t forget to properly mark snapshots and servers to exclude their usage in the future. We would recommend never to use those snapshots and servers. If you still need to use the SolarWinds application you need to rebuild SolarWinds servers from backups.
- Since SolarWinds has a backdoor we can’t be sure that all activities inside the servers with SolarWinds were discovered by researchers. We probably can see more and more versions of what SolarWinds backdoor could do on the servers. That is why any usage of infected SolarWinds servers should be prohibited. They can be used only for research and investigation purposes.
- Check if you have installed the SolarWinds application in the past. It is still possible to have a situation when you did an assessment of the SolarWinds application, installed it on your server, configured it to monitor your servers, and decided not to continue with SolarWinds. In a good situation after all of that, you just removed SolarWinds server and that is all. Even if you don’t use SolarWinds now but had been using it after March 2020 you should consider that your server was compromised and do all things as if you had it installed right now. And so, check out your historical notes. Talk to the teams. Maybe someone just explored different network monitoring options as an afterthought without a single record of it?
- If you can’t turn off the server then you need to block any inbound and outbound connections except allowed ones.
2. Analyze stored network traffic/audit logs for
- Indications of compromise. FireEye published a list of IPs and domains that belong to attackers. It is a good point to start searching them in your network flow logs and DNS logs. Later LabVerge will publish Splunk Searches for Paloalto Firewall logs, AD DNS logs, and OpenDNS. Keep in mind that you need to search for events from the time when SolarWinds vulnerable version was installed. So, March 2020 when the vulnerable version was published for public access is the date when you need to start to search. Probably searches will take some time because the time frame is big.
- Create alerts in your SIEM on the indication of compromise – DNS and IP addresses and hashes. So, you would know if somebody from your network still communicates with the Command and Control server (C2). You should know that backdoor implements time threshold checks to ensure that there are unpredictable delays between C2 communication attempts, further frustrating traditional network-based analysis.
- Check if your security firewalls, endpoint protection, and other security controls are up to date and contain an indication of compromised IPs and DNS. Try to connect to those IP’s under your security protection contact and check if it was caught and blocked.
- Read newsletters from all your security vendors, they probably published announcements about how they prevent/react to this attack. Some of them can say that you are safe because malware didn’t start when their protection was running. And yes, it can be true because malware has detected such kinds of things and tried to deactivate them.
- Check all new external DNS domains to which a small number of your hosts have had connections. Just as a proactive measure with deep-level analysis of traffic.
- Check any logins to the Solarwinds server. All used credentials can be compromised and should be rotated.
- For a case, if you are not sure about how many SolarWinds servers you have, you need to check your network logs to solarwinds.com. Who knows maybe you still have some enabled tests server In the far corner of your network.
- If you have enabled TLS decryption that you probably can search hashed from IoC https://github.com/fireeye/sunburst_countermeasures/blob/main/indicator_release/Indicator_Release_Hashes.csv
- Even if it is not enabled you can try to find those hashes in your SIEM and Endpoint protection tools.
- Check outbound RDP connections from SolarWinds servers. Figure out if they are allowed or not. It probably can be malware activity for lateral movement.
3. Review public evidence list
- Sunburst would send the data it collected from an infected network to a C2 server URL that was a unique per victim. This unique URL was a subdomain for avsvmcloud[.]com and contained four parts, where the first part contained the encoded name of the victim’s local network domain. Public searchers sifting through historical web traffic and passive DNS data to collect information on traffic going to the avsvmcloud[.]com domain, crack the subdomains, and then track down companies that installed a trojanized SolarWinds Orion app. List of domains suspected to be attacked by Sunburst https://pastebin.com/raw/6NukuxBN https://twitter.com/RedDrip7/status/1339168187619790848. Try to search there for the name of your company, your AD. Note: It is not a 100% accurate list and it can be used as a secondary source of evidence.
4. Rotate credentials. It includes the following list, and possibly more
- A service account used for the Orion platform.
- Credentials of user’s accounts connected to SolarWinds servers.
- A service account used to access the SQL database (possibly the same as above, or a SQL login)
- Domain-based accounts used for WMI monitoring Windows servers and applications.
- An account with an Exchange Server mailbox for sending alerts.
- SNMP community strings and/or SNMPv3 credentials.
- Application credentials.
- Hypervisor credentials in NPM, VMAN, or SAM.
- SSH credentials in NCM.
5. Audit network device configurations
- Based on your changed management model you need to review all changes in network devices performed after March 2020.
- Firmware and software validation. Do hash verification against known good versions of firmware.
6. File system analysis:
- SUNBURST stores configuration inside legitimate SolarWinds.Orion.Core.BusinessLayer.dll.config file. Two existing settings in the appSettings section: ReportWatcherRetry and ReportWatcherPostpone. If ReportWatcherRetry is 3 then malware has been deactivated and will no longer perform any network activity. If you have other values than SUNBURST were probably active and communicated to C&C.
- Check Windows Tasks Manager to detect ongoing tasks related to SUNBURST activity
- Check a status of an endpoint protection application. SUNBURST doesn’t want to be discovered and tries to disabled endpoint protection on servers. Check registry value HKLM\SYSTEM\CurrentControlSet\services\<service_name>\Start if value 4 then SUNBURST disable this service on the next power cycle. Find when last time your server was rebooted. After the registry modification is made, SUNBURST updates the ReportWatcherPostpone configuration value to reflect the service it disabled. Then, the backdoor pauses and retries the process and service blocklist checks at a later time. Subsequent service blocklist checks skip services already present in the ReportWatcherPostpone configuration key. SUNBURST will not treat the services it has disabled as members of the blocklist anymore. Therefore, during an incident response, forensic teams should consider recovering and decoding this configuration key to parse out which services SUNBURST attempted to disable.
- Check your hard drive file “gracious_truth.jpg” which belongs to the TEARDROP memory only dropper. If it exists then probably SUNBURST was active and malicious files were uploaded to the server.
- The existence of the file C:\WINDOWS\SysWOW64\netsetupsvc.dll may also indicate a compromise
NOTE: This backdoor is smart and can revert its own changes. So, If you have Windows Shadowing or different in time backups then you probably can restore the malware configuration SolarWinds.Orion.Core.BusinessLayer.dll.config and other files.