... InfluxDB is going back to SQL with Flight in I0X - but the greatness of Flux lives on!
💡 Fluxpipe is a stand-alone, serverless Flux API/Pipeline for querying, analyzing and interacting with data on the cloud. TLDR: Flux scripts outside of InfluxDB!
❓ New to Flux? Catch up in less than 3 minutes!
Fluxpipe is a service emulating the InfluxDB API compatible with any existing API clients as well as data sources such as Grafana, but that's only the beginning!
Fluxpipe is also designed to unleash Flux scripts within serverless functions at lightning speeds, with a minuscule footprint compared to its direct competition.
TLDR; Here's a Live Demo
💡 Fluxpipe is built and tested against the latest stable Flux release, and includes several contrib modules designed for qryn.cloud and clickhouse interoperability.
Lambda Flux
This article describes how to:
Create and test an AWS Lambda function for Fluxpipe.
Use Fluxpipe Gigapipe to query any supported data source.
Use Flux scripts to filter, process and downsample the data.
Write the data back to qryn, InfluxDB or any supported data source.
Use CloudWatch or EventBridge to trigger your functions automatically.
Create an AWS Lambda Function
To create an AWS Lambda function, log into your AWS console.
➡ Search for AWS Lambda and select the service. Then, click Create Function.
➡ Choose Container Image and use the public Fluxpipe ECR instance URI.
Push the latest fluxpipe-lambda container to your ECR storage for best results
➡ Click Create Function at the bottom right when you’re done.
Validation
Let's test our new Fluxpipe Lambda using a Flux script.
The Lamba expects JSON requests with a query key:
{
'query': 'import g "generate" g.from(start: 2022-04-01T00:00:00Z, stop: 2022-04-01T00:03:00Z, count: 5, fn: (n) => n+1)'
}
And the response would look like this:
#datatype,string,long,dateTime:RFC3339,long
#group,false,false,false,false
#default,_result,,,
,result,table,_time,_value
,,0,2022-04-01T00:00:00Z,1
,,0,2022-04-01T00:00:36Z,1
,,0,2022-04-01T00:01:12Z,1
,,0,2022-04-01T00:01:48Z,1
,,0,2022-04-01T00:02:24Z,1
Ready
Once your Lambda is validated, sky's the limit! Move your existing Influx/Flux workflows into your lambdas and use them to downsample, forecast and report.
💡 For additional security, use AWS Secrets Manager and ENV variables to control sensible fields (such as authentication ids, tokens, etc) in your Flux scripts.
Automation
If you’re looking to perform a downsampling task, you’ll need to run your Lambda script on a schedule. You can use CloudWatch or EventBridge to create a rule and target your AWS Lambda function to run Flux scripts on a user-defined schedule.
Use the following documentation depending on your preferred service:
Conclusion
Just a few clicks and you are all set with a low-cost, high-power Fluxpipe Lambda function ready to perform simple and complex data processing tasks at any scale.
