As Elasticians, we have the opportunity to observe various Elastic use cases. As former SOC analysts, we find ourselves drawn to the security use cases. We are always looking for ways to leverage the Elastic Stack to add value to the Elastic and cybersecurity community. We are constantly researching — sifting through news articles, white papers, databases, etc. We realized that we were spending a lot of time visiting the same sites and needed to consolidate the feeds (duh!).
Below, we’ll break down how we used RSS feeds, Logstash, Elasticsearch, and Kibana to ingest, consolidate, aggregate, visualize, and search cybersecurity content of interest — and how you can do the same.
[Related article: Cybersecurity is a data challenge, and better search technology is key to improving visibility and action]
RSS feeds
There are several ways to consolidate RSS feeds. Many feeders and aggregators are subscription based, but we feel that content curation should be free. Our initial impulse was, “let’s write a Python script, send the data to Logstash, watch it flow into Elasticsearch, and observe via Kibana…” As we started planning and researching, we encountered a solution that would allow us to bypass the Python script entirely: the RSS input plugin for the Logstash pipeline. At that point, we had identified all of the necessary ingredients to start building an Elastic-based Open Source Intelligence Tool (OSINT).
Our project’s name is A Quick RSS Cybersecurity News Feed. These are its components:
- RSS Feeds
- Logstash configuration(s)
- Elasticsearch configuration(s)
- Kibana Visualizations and Dashboards
Leave a Reply