Flux
Flux is an open source functional data scripting language from InfluxDB designed for querying, analyzing, and acting on data. Flux supports multiple data source types, including Time series databases (such as InfluxDB), Relational SQL databases (such as MySQL, Postgres or ClickHouse), HTTP APIs and CSV datasources.
Flux unifies querying, processing, writing, and acting on data into a single syntax. The flux language is designed to be usable, readable, flexible, composable, testable, contributable, and shareable. You can even extend flux using flux scripts!
Flux is part of InfluxDB 1.7 and 2.0, but can be run independently of those.
Unfortunately, no stand-alone version of Flux existed in the wild .... until now!
Hello, FluxPipe
FluxPipe is an experimental stand-alone Flux API built for serverless and used by our tools at qxip/metrico. Made in rust/golang it offers a thin API layer (emulating a basic InfluxDB 2.x endpoint) and near instant startup and query processing times.
Just like we did for ClickHouse the Fluxpipe binary is built to portable, static and ready to run anywhere - without storage - instantly compatible with most Flux scripts and ready to use and modify for an endless variety of data piping and processing purposes.
What can we do with our fresh portable Flux binary?
Of course you can run it on your servers large or small. But that's so boring!
Flux into Space
Today, we're launching Flux into Space with Deta 🚀
Warnings
This is a free experimental ride for demo purposes. Not a final product!
The goal is to showcase the flexibility of Deta and power of Influx/Flux
Sign up for Deta Space Alpha
Deta is your own personal computer in the cloud: private, secure & always online
🟣 All apps on Deta Space run in your own sandboxed 'personal cloud'
🟣 App developers cannot see or touch your personal infrastructure
🟣 App users run clones and remain in the drivers seat with all the data
Deta Space Micros, Base and Drive are perfect for modern web applications.
But can they run something as complex as Flux? TLDR; HELL YEAH!
Launch your Flux Space
Ready? Browse to the Flux Space page using Discovery and Install
Space will generate a unique secure URL for your Flux Space API endpoint
https://FLUX-SPACE-URL.deta.app
Awesome! Your serverless, stateless and bottomless Flux API is ready! 🔥
TLDR; test before reading using the preset selector
📙 Flux Language
FluxPipe is designed to serve and execute Flux scripts for querying, analyzing, and acting on data. Flux scripts can range from extremely simple, to extremely complex, importing standard and custom functions available to extend the scope of execution.
Let's try it out!
You can query your Flux endpoint using CURL or any Flux client such as Grafana
CURL
POST your fist query with curl to generate some sample data.
curl -XPOST https://FLUX-SPACE-URL.deta.app/api/v2/query -sS \
-H 'Accept:application/csv' \
-H 'Content-type:application/vnd.flux' \
-d 'import g "generate" g.from(start: 2022-04-01T00:00:00Z, stop: 2022-04-01T00:03:00Z, count: 3, fn: (n) => n)'
Congratulations! You're now a Flux Space Cadet! 🏆🚀
Grafana
There's more! You can configure your Flux Space endpoint as a new InfluxDB datasource
Save and Explore data using your own serverless datasource!
The world is your data oyster! Let's start exploring some simple scrips.
Query MySQL
import "sql"
sql.from(driverName: "mysql",dataSourceName: "default:password@tcp(127.0.0.1:9004)/system", query: "SELECT database, total_rows FROM tables")
|> rename(columns: {database: "_value", total_rows: "_data"}) |> keep(columns: ["_value","_data"])
Query APIs
import "experimental/http/requests"
import ejson "experimental/json"
import "json"
import "array"
response =
requests.post(
url: "https://goolnk.com/api/v1/shorten",
body: json.encode(v: {url: "http://metrico.in"}),
headers: ["Content-Type": "application/json"],
)
data = ejson.parse(data: response.body)
array.from(rows: [data])
Scrape Metrics
import "experimental/prometheus"
prometheus.scrape(url: "http://prometheus-exporter:9096/metrics")
... and there's so much more you can do using the excellent flux documentation
Now, you might know we're ClickHouse freaks around here so...
Flux 💛 ClickHouse
Let's try a simple ClickHouse query via HTTP API + CSVWithNames format
import "csv"
import "experimental/http/requests"
import "strings"
url = "https://clickhouse.server"
query = "SELECT name, value FROM table FORMAT CSVWithNames"
response = requests.get(url:url, params: ["query": [query]])
csv.from(csv: string(v: response.body), mode: "raw")
That's clean and simple. Works anywhere.
Fluxpipe offers an experimental ClickHouse binary sql driver (PR open)
import "sql"
sql.from(
driverName: "clickhouse",
dataSourceName: "clickhouse://default:@clickhouse-host:9000/system",
query: "SELECT database, total_rows FROM tables WHERE total_rows > 0"
)
|> rename(columns: {database: "_value", total_rows: "_data"})
|> keep(columns: ["_value","_data"])
Sharing Data
Let's use S3rver-Deta Drive as shared CSV storage between Flux and ClickHouse
Insert some data in an S3 Table using ClickHouse
INSERT INTO FUNCTION s3('S3RVER-SPACE-ENDPOINT.deta.app/test/flux.csv', 'CSV', 'name String, value UInt32') FORMAT Values ('one', 1), ('two', 2),('three', 3), ('four', 4), ('five',5);
Query id: ad61bcbc-a92b-4e55-8f71-2063d5c9488c
Ok.
5 rows in set. Elapsed: 0.495 sec.
Let's read our data back from S3 and parse it using Flux
import "csv"
import "experimental"
import "experimental/http/requests"
response = requests.get(
url: "https://S3RVER-SPACE-ENDPOINT.deta.app/test/flux.csv",
)
csv.from(csv: string(v: response.body), mode: "raw")
Et Voila' - works like magic!
Pinch me awake! Is this is all powered by serverless, on-demand functions?! 🤯
Technical Notes
⚠️ Fluxpipe does not support internal buckets - only external datasources
⚠️ Fluxpipe does not support secrets - use ENV variables instead
⚠️ Certain scripts might fail - please open an issue and we'll take a look
Conclusion
This Flux server does not exist. It's a serverless Grafana datasource from Space!
The question is: What will you build with it? 🚀 🚀 🚀
Thanks to the Deta Team for their dedication and for letting us play with their alpha!
▶ UPDATE
- Fluxpipe can be deployed on [fly.io] and any other service supporting Docker
- Fluxpipe now has a Playground UI so you don't need Grafana to try this out!
Want More?
Did this get you excited? Join our team, we're hiring!
Working with ClickHouse? Try our polyglot observability stack
Enjoy and Share your Comments! 👋