S3 Bucket Lambda Function

January 28, 2020
Andrew Konoff
I'm the product manager on the Integrations squad. Previously at Heroku, Realm, and Salesforce. Big fan of writing songs and making tiki drinks.

AWS has a lot of services, and they all generate logs. A lot of logs. We’ve worked hard to make sure you can capture logs from every source and service on AWS, and today we're happy to announce the final piece of our AWS logging puzzle: LogDNA's S3 Collector integration. It's an easy-to-use Lambda function that lets you ingest any AWS logs that get dumped to S3 bucket – like logs from CloudFront and ELB. Whatever the log, no matter where it comes from on AWS, you can now get it into LogDNA.

If you use Amazon’s various container platforms, like Elastic Kubernetes Service (EKS) and Elastic Container Service (ECS), our LogDNA Agent is there for you. You can also use our LogSpout integration on ECS, if the Agent isn’t convenient or if you’d like a more Docker-focused experience.

But what about all the seeming hundreds of other services that you can use on AWS? Until now, you had to use AWS CloudWatch to aggregate them, and then integrate CloudWatch with LogDNA to send them over to us. That can start to feel like overkill if you’ve just got a few things on AWS, or redundant when you want to keep your log alerting and observation happening in LogDNA.

Introducing LogDNA’s S3 Log Integration to export Lambda logs to S3

That’s why we’re really happy to offer a lightweight S3 Collection. First, set up your AWS services to dump their logs to the S3 bucket. Then, you set up our Lambda function, specifying which buckets it should be triggered by. Then anytime new log files get added to those buckets, it’ll send them to LogDNA. It’s easy to set up, highly scalable, and super affordable to run. To get started, just follow the docs here.

We’ve added better parsing of AWS logs too so that your CloudFront and ELB/ALB logs are precisely parsed and searchable. These extremely long ALB logs, for instance, suddenly become a lot easier to understand and query:

For the list of supported parsing types, check out this link. We’re working on adding native parsing for more AWS services, so just let us know what you’d like to see!

Our other AWS Integrations

If you’d like to discover the right way for you to integrate AWS’ many services into LogDNA, just jump into our docs. A few highlights:

Are we missing anything? Let us know by Twitter, or at betafeedback@logdna.com.

Read More About: 

30% Discount for 6 Months Without Any Annual Contract

Get Started Today

Logging in the Age of DevOps eBook

Download Now