FluxPipe: Serverless Flux / InfluxDB

FluxPipe: Serverless Flux / InfluxDB

Launching Flux into Space!

ยท

5 min read

Play this article

image.png

Flux

Flux is an open source functional data scripting language from InfluxDB designed for querying, analyzing, and acting on data. Flux supports multiple data source types, including Time series databases (such as InfluxDB), Relational SQL databases (such as MySQL, Postgres or ClickHouse), HTTP APIs and CSV datasources.

Flux unifies querying, processing, writing, and acting on data into a single syntax. The flux language is designed to be usable, readable, flexible, composable, testable, contributable, and shareable. You can even extend flux using flux scripts!

Flux is part of InfluxDB 1.7 and 2.0, but can be run independently of those.

Unfortunately, no stand-alone version of Flux existed in the wild .... until now!

Hello, FluxPipe

FluxPipe is an experimental stand-alone Flux API built for serverless and used by our tools at qxip/metrico. Made in rust/golang it offers a thin API layer (emulating a basic InfluxDB 2.x endpoint) and near instant startup and query processing times.

Just like we did for ClickHouse the Fluxpipe binary is built to portable, static and ready to run anywhere - without storage - instantly compatible with most Flux scripts and ready to use and modify for an endless variety of data piping and processing purposes.

What can we do with our fresh portable Flux binary?

Of course you can run it on your servers large or small. But that's so boring!


image.png

Flux into Space

Today, we're launching Flux into Space with Deta ๐Ÿš€


Warnings
This is a free experimental ride for demo purposes. Not a final product!
The goal is to showcase the flexibility of Deta and power of Influx/Flux


Sign up for Deta Space Alpha

Deta is your own personal computer in the cloud: private, secure & always online

๐ŸŸฃ All apps on Deta Space run in your own sandboxed 'personal cloud'
๐ŸŸฃ App developers cannot see or touch your personal infrastructure
๐ŸŸฃ App users run clones and remain in the drivers seat with all the data

Deta Space Micros, Base and Drive are perfect for modern web applications.

But can they run something as complex as Flux? ย  TLDR; HELL YEAH!


Launch your Flux Space

Ready? Browse to the Flux Space page using Discovery and Install

Space will generate a unique secure URL for your Flux Space API endpoint

https://FLUX-SPACE-URL.deta.app

โ˜๏ธโ˜๏ธโ˜๏ธ This is a Live demo, not just an image โ˜๏ธโ˜๏ธโ˜๏ธ

Awesome! Your serverless, stateless and bottomless Flux API is ready! ๐Ÿ”ฅ

TLDR; test before reading using the preset selector


๐Ÿ“™ Flux Language

FluxPipe is designed to serve and execute Flux scripts for querying, analyzing, and acting on data. Flux scripts can range from extremely simple, to extremely complex, importing standard and custom functions available to extend the scope of execution.

Let's try it out!

You can query your Flux endpoint using CURL or any Flux client such as Grafana

CURL

POST your fist query with curl to generate some sample data.

curl -XPOST https://FLUX-SPACE-URL.deta.app/api/v2/query -sS \
  -H 'Accept:application/csv' \
  -H 'Content-type:application/vnd.flux' \
  -d 'import g "generate" g.from(start: 2022-04-01T00:00:00Z, stop: 2022-04-01T00:03:00Z, count: 3, fn: (n) => n)'

Congratulations! You're now a Flux Space Cadet! ๐Ÿ†๐Ÿš€

Grafana

There's more! You can configure your Flux Space endpoint as a new InfluxDB datasource

image.png

Save and Explore data using your own serverless datasource!

image.png


The world is your data oyster! Let's start exploring some simple scrips.

Query MySQL

import "sql" 

sql.from(driverName: "mysql",dataSourceName: "default:password@tcp(127.0.0.1:9004)/system", query: "SELECT database, total_rows FROM tables") 
|> rename(columns: {database: "_value", total_rows: "_data"}) |> keep(columns: ["_value","_data"])

Query APIs

import "experimental/http/requests"
import ejson "experimental/json"
import "json"
import "array"

response =
     requests.post(
         url: "https://goolnk.com/api/v1/shorten",
         body: json.encode(v: {url: "http://metrico.in"}),
         headers: ["Content-Type": "application/json"],
     )

data = ejson.parse(data: response.body)
array.from(rows: [data])

Scrape Metrics

import "experimental/prometheus"
prometheus.scrape(url: "http://prometheus-exporter:9096/metrics")

... and there's so much more you can do using the excellent flux documentation

image.png


Now, you might know we're ClickHouse freaks around here so...

Flux ๐Ÿ’› ClickHouse

Let's try a simple ClickHouse query via HTTP API + CSVWithNames format

import "csv"
import "experimental/http/requests"
import "strings"

url = "https://clickhouse.server"
query = "SELECT name, value FROM table FORMAT CSVWithNames"
response = requests.get(url:url, params: ["query": [query]])

csv.from(csv: string(v: response.body), mode: "raw")

That's clean and simple. Works anywhere.

Fluxpipe offers an experimental ClickHouse binary sql driver (PR open)

import "sql" 

sql.from(
  driverName: "clickhouse",
  dataSourceName: "clickhouse://default:@clickhouse-host:9000/system",
  query: "SELECT database, total_rows FROM tables WHERE total_rows > 0"
) 
|> rename(columns: {database: "_value", total_rows: "_data"})
|> keep(columns: ["_value","_data"])

Sharing Data

Let's use S3rver-Deta Drive as shared CSV storage between Flux and ClickHouse

Insert some data in an S3 Table using ClickHouse

INSERT INTO FUNCTION s3('S3RVER-SPACE-ENDPOINT.deta.app/test/flux.csv', 'CSV', 'name String, value UInt32') FORMAT Values ('one', 1), ('two', 2),('three', 3), ('four', 4), ('five',5);

Query id: ad61bcbc-a92b-4e55-8f71-2063d5c9488c

Ok.

5 rows in set. Elapsed: 0.495 sec.

Let's read our data back from S3 and parse it using Flux

import "csv"
import "experimental"
import "experimental/http/requests"
response = requests.get(
    url: "https://S3RVER-SPACE-ENDPOINT.deta.app/test/flux.csv",
)
csv.from(csv: string(v: response.body), mode: "raw")

image.png

Et Voila' - works like magic!


Pinch me awake! Is this is all powered by serverless, on-demand functions?! ๐Ÿคฏ

Technical Notes

โš ๏ธ Fluxpipe does not support internal buckets - only external datasources
โš ๏ธ Fluxpipe does not support secrets - use ENV variables instead
โš ๏ธ Certain scripts might fail - please open an issue and we'll take a look

Conclusion

This Flux server does not exist. It's a serverless Grafana datasource from Space!

The question is: What will you build with it? ๐Ÿš€ ๐Ÿš€ ๐Ÿš€

Thanks to the Deta Team for their dedication and for letting us play with their alpha!

โ–ถ UPDATE

  • Fluxpipe can be deployed on [fly.io] and any other service supporting Docker
  • Fluxpipe now has a Playground UI so you don't need Grafana to try this out!

โ˜๏ธโ˜๏ธโ˜๏ธ This is a Live demo, not just an image โ˜๏ธโ˜๏ธโ˜๏ธ

Want More?

Did this get you excited? Join our team, we're hiring!
Working with ClickHouse? Try our polyglot observability stack

Enjoy and Share your Comments! ๐Ÿ‘‹

Did you find this article valuable?

Support qryn by becoming a sponsor. Any amount is appreciated!

ย