Regular Expressions in Grep

Read more
industry

REQUIREMENTS
LOGDNA SOLUTIONS
BUSINESS IMPACT

case studies

LogDNA Helps Employment Hero Embrace Microservices at Scale

Employment Hero, which was founded in 2014, originally deployed a software stack composed of monolithic applications. It started refactoring those applications to fit a microservices architecture around 2017. At the same time, it made the move toward a Kubernetes-centric environment for hosting its applications.The company’s engineers quickly discovered, however, that the logging and monitoring solution they had in place at the time was not sufficient for managing microservices applications. The platform did not offer an easy way to collect logs within a distributed environment where each service stores logs in a different place, let alone analyze the logs efficiently. Tracing user requests across the logs of multiple services was difficult, and there was no support for live tailing logs to track the latest activity.

As Luong Vo, Platform Engineering Manager atEmployment Hero, explained, “We needed the logging system to be scalable, easy to search, accurate in time, and support live tailing,” and the existing logging solution didn’t do that. At first, Luong and his team experimented with using self-hosted Elasticsearch, Logstash and Kibana—theELK stack—to meet their need for microservices friendly log aggregation and analytics. However, “we found that we were spending too much time scaling Elasticsearch and maintaining the whole stack,” he said, prompting them to search for a more user friendly alternative.

After evaluating other logging platforms, theEmployment Hero team settled on LogDNA.

Read more
Better Mortgage Speeds Kubernetes Troubleshooting with Help from LogDNA

Better Mortgage promises its customers a seamless experience for securing a mortgage. To deliver fully on that promise, the company must ensure that it can quickly identify, troubleshoot, and resolve software performance and security issues that arise within its IT environments.

With the help of LogDNA, Better Mortgage was able to build a software monitoring and analytics stack that delivers the visibility its engineers need to find and fix problems fast. Read the full case study for a look at why Better Mortgage migrated from an open source analytics solution to LogDNA, and how LogDNA helped the company optimize the management of its complex, Kubernetes-based environments.

Read more
LogDNA is the Key to Kubernetes Observability

When you develop security software for containers, Kubernetes, and cloud services, as Sysdig does, you have an especially keen appreciation of what it takes to build an effective observability stack. You are also in a strong position to appreciate the efficiency that comes with modern observability tools that work seamlessly with next-generation platforms, such as Kubernetes. 

Both of these factors contributed to Sysdig’s decision to adopt LogDNA for gaining operational observability into its internal IT infrastructure. We recently spoke to Mark Breitung, a senior member of the DevOps team at Sysdig, about why the company chose LogDNA, and how the solution enables the holistic, contextualized observability that his team needs to keep the Sysdig platform running smoothly.

Read more
Modern Logging for Modern Account Opening

We sat down with James Qualls, a Director of Engineering at MANTL, to hear how LogDNA is empowering the developers on his team to own their monitoring. James is one of two Directors of Engineering at MANTL. He handles Infrastructure, and his counterpart, Eric Gomez, handles Application Architecture. James and his team are responsible for everything underneath the work that Eric’s team stands on top of.

Read more
LogDNA and IBM find synergy in cloud

In 2018 LogDNA partnered with tech giant, IBM, to become the sole logging provider for IBM Cloud. This is the story of how the partnership started. Since then, LogDNA has grown to support thousands of internal IBM teams and enterprises on IBM Cloud and is deployed in eight multi-zone regions across the globe. 

Users have access to two offerings in the cloud catalog. IBM Log Analysis with LogDNA provides detailed insights into all environments by centralizing application, system, and server logs into a single UI. Users can deploy on IBM Cloud in minutes, see their logs instantly in Live Tail, and search using natural language. Intelligent Parsing, real-time Alerts, and custom visualizations help teams stay up to date with the data that’s most valuable to them. 

IBM Cloud Activity Tracker with LogDNA is for anyone using IBM Cloud Services to help them gain insights into actions that change the state of these services. This offering provides real-time Alerts, custom Views, and data visualizations with Boards and Graphs to show a snapshot of Cloud Service activities and help users identify trends over time.

Read more

If you’ve ever had to search, parse, or edit blocks of text programmatically, chances are you’re familiar with regular expressions (also known as regex or regexp). Let’s cover exactly what Regular Expressions are, what they’re used for, benefits, and some examples.

What are Regular Expressions?

Regex is a special text string/language used for describing search patterns and matching strings in text. Its flexible and powerful syntax lets you create detailed search patterns, from simple words and phrases to complex constructs like e-mail addresses and phone numbers. It’s much more powerful than a simple string comparison, and is almost universally supported across programming languages, frameworks, and text editors.

Linux comes with GNU grep command which supports regex. Grep stands for “global regular expression print”. Grep is used to find what you’re looking for, stored anywhere in the file system matching a specified pattern.

Simple grep examples:

grep ‘word’ file1 file2 file3

grep ‘username’ /etc/passwd

You can use regex to specify a string of characters or pattern for grep to match instead of words.

Benefits of Regex

Regexes are much more flexible than traditional text searches. They can detect almost any pattern of letters, numbers, symbols, special characters, and even metacharacters. Where traditional searches look for exact matches, regexes can match patterns of varying length. This makes them useful for finding constructs such as email addresses, IP addresses, URLs, and phone numbers.

Regexes are also concise. A single regex string can contain multiple search terms, perform multiple operations, and return multiple matches. This makes them very easy to implement, reuse, and modify.

Limitations of Regex

Regex has a steep learning curve. Even basic regular expressions are difficult to break down into their base operations. Compared to verbose languages like Python, understanding a regex requires a detailed understanding of the language. This can make expressions difficult to troubleshoot, especially for beginners. This is best expressed in the famous quote by Jamie Zawinski:

Some people, when confronted with a problem, think ‘I know, I’ll use regular expressions.’ Now they have two problems.

Heavy text processing can also be slow, depending on the complexity of the expression and the amount of text to search. There are ways to improve performance, such as using compiled expressions (the default in languages like Python), but it ultimately comes down to the efficiency of the expression.

How is RegEx Used? Use Cases and Examples:

Regex has a number of use cases, including:

Searching

Regex is designed for searching. Traditional search methods might only let you search for a specific string, but regex offers much more flexibility and control over how searches are performed.

Example:

Imagine you have a text document (such as a log file) and you want to find all instances of an email address appearing within the document. How would you go about this? You could start by searching for the “@” character, or for “.com”, but what if the document also includes Twitter handles or website URLs? What about email addresses that end in “.edu”, or “.net”? You would likely need to run multiple searches at a time and use complex string manipulation rules to extract out each potential match.

Alternatively, you could create a single regex expression that searches specifically for email addresses. One method is to use the following expression:

[a-zA-Z0-9-.!#$%&'*+\/=?^_`{;}~]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9]{2,3}

Let’s break down this expression:

[a-zA-Z0-9-.!#$%&'*+\/=?^_`{;}~]+Match any number of letters, numbers, or special characters listed.@Match the “@” symbol.[a-zA-Z0-9-]+Match any number of letters, numbers, or hyphen characters.\.Match a period.[a-zA-Z0-9]{2,3}Match any two or three letter word containing letters or numbers.

With this expression, we can return all instances of “user@example.com”, “user.name@123company.co“, or even “super_user+$10k@dash-co.net”, but not “@example” or “http://example.com“.

User Input Validation

Regex is often used as an input validation tool. Imagine you have a website where users can sign up by providing their email address. Before the registration can be completed, the user’s email address must be verified. With regex, we can perform a simple validation test that checks the formatting of the user’s address before we allow them to register. We can even use JavaScript to perform this test and notify the user in real time, while using the same expression used in the previous example.

String Replacement and Masking

We discussed how regex can be used to find patterns of text within larger documents. But what if you wanted to replace, mask, or delete certain text?

Example:

Consider a payment processing service that occasionally logs sensitive data such as credit card numbers and bank account details. To protect their users’ privacy, the service should automatically scrub this data before sending its logs to a centralization service. But how do we detect and erase this data after the log has already been written?

With regex, we can create expressions to detect numbers matching the formats used by credit card vendors. We can then use a method like Python’s re.sub() to substitute each instance with another value.

Using Regex in LogDNA’s Stream Editor

Log messages don’t always appear perfectly formatted. This is why the LogDNA web app includes a stream editor feature that lets you change the formatting of your log data in real-time. You can use a regular expression as your search term, as well as toggle case sensitivity and global searching. This works similar to the sed command, while also formatting live log data.

Example:

Imagine you have an application that writes multiline logs to syslog. To avoid generating multiple syslog events from a single application event, the syslog service automatically escapes newline characters. This ensures each event only writes a single syslog message, but this makes the log stream appear cluttered and difficult to read. With LogDNA, we can use the search and replace feature to find and replace all instances of the escaped newline character with an actual newline character:

The “i” button toggles case sensitivity for the regular expression, while the “g” button toggles global or local matching. If global matching is disabled, only the first match in the stream is replaced. Clicking on the check mark performs the replace, and clicking on the “x” reverts it. Now, any current and new syslog messages will be displayed over multiple lines while leaving the actual log data untouched.

Conclusion

Despite being almost thirty years old, regex is still unfamiliar and esoteric territory for many developers. However, its flexibility and ubiquity make it a valuable addition to any developer’s toolkit. If you want to learn more about regex or practice creating different expressions, sites like RegExr and regex101 provide interactive editors. Regular-Expressions.info also provides detailed tutorials, examples, and quick start guides.

Read more
industry

REQUIREMENTS
LOGDNA SOLUTIONS
BUSINESS IMPACT

case studies

LogDNA Helps Employment Hero Embrace Microservices at Scale

Employment Hero, which was founded in 2014, originally deployed a software stack composed of monolithic applications. It started refactoring those applications to fit a microservices architecture around 2017. At the same time, it made the move toward a Kubernetes-centric environment for hosting its applications.The company’s engineers quickly discovered, however, that the logging and monitoring solution they had in place at the time was not sufficient for managing microservices applications. The platform did not offer an easy way to collect logs within a distributed environment where each service stores logs in a different place, let alone analyze the logs efficiently. Tracing user requests across the logs of multiple services was difficult, and there was no support for live tailing logs to track the latest activity.

As Luong Vo, Platform Engineering Manager atEmployment Hero, explained, “We needed the logging system to be scalable, easy to search, accurate in time, and support live tailing,” and the existing logging solution didn’t do that. At first, Luong and his team experimented with using self-hosted Elasticsearch, Logstash and Kibana—theELK stack—to meet their need for microservices friendly log aggregation and analytics. However, “we found that we were spending too much time scaling Elasticsearch and maintaining the whole stack,” he said, prompting them to search for a more user friendly alternative.

After evaluating other logging platforms, theEmployment Hero team settled on LogDNA.

Read more
Better Mortgage Speeds Kubernetes Troubleshooting with Help from LogDNA

Better Mortgage promises its customers a seamless experience for securing a mortgage. To deliver fully on that promise, the company must ensure that it can quickly identify, troubleshoot, and resolve software performance and security issues that arise within its IT environments.

With the help of LogDNA, Better Mortgage was able to build a software monitoring and analytics stack that delivers the visibility its engineers need to find and fix problems fast. Read the full case study for a look at why Better Mortgage migrated from an open source analytics solution to LogDNA, and how LogDNA helped the company optimize the management of its complex, Kubernetes-based environments.

Read more
LogDNA is the Key to Kubernetes Observability

When you develop security software for containers, Kubernetes, and cloud services, as Sysdig does, you have an especially keen appreciation of what it takes to build an effective observability stack. You are also in a strong position to appreciate the efficiency that comes with modern observability tools that work seamlessly with next-generation platforms, such as Kubernetes. 

Both of these factors contributed to Sysdig’s decision to adopt LogDNA for gaining operational observability into its internal IT infrastructure. We recently spoke to Mark Breitung, a senior member of the DevOps team at Sysdig, about why the company chose LogDNA, and how the solution enables the holistic, contextualized observability that his team needs to keep the Sysdig platform running smoothly.

Read more
Modern Logging for Modern Account Opening

We sat down with James Qualls, a Director of Engineering at MANTL, to hear how LogDNA is empowering the developers on his team to own their monitoring. James is one of two Directors of Engineering at MANTL. He handles Infrastructure, and his counterpart, Eric Gomez, handles Application Architecture. James and his team are responsible for everything underneath the work that Eric’s team stands on top of.

Read more
LogDNA and IBM find synergy in cloud

In 2018 LogDNA partnered with tech giant, IBM, to become the sole logging provider for IBM Cloud. This is the story of how the partnership started. Since then, LogDNA has grown to support thousands of internal IBM teams and enterprises on IBM Cloud and is deployed in eight multi-zone regions across the globe. 

Users have access to two offerings in the cloud catalog. IBM Log Analysis with LogDNA provides detailed insights into all environments by centralizing application, system, and server logs into a single UI. Users can deploy on IBM Cloud in minutes, see their logs instantly in Live Tail, and search using natural language. Intelligent Parsing, real-time Alerts, and custom visualizations help teams stay up to date with the data that’s most valuable to them. 

IBM Cloud Activity Tracker with LogDNA is for anyone using IBM Cloud Services to help them gain insights into actions that change the state of these services. This offering provides real-time Alerts, custom Views, and data visualizations with Boards and Graphs to show a snapshot of Cloud Service activities and help users identify trends over time.

Read more

Get started with LogDNA

Start Trial