Serverless Node.js URL Shortener App powered by Upstash Kafka and Materialize

App structure

The demo app has the following structure:

  • A serverless Cloudflare Worker that lets you add short links and redirect them to other URLs.
  • All data is stored in Upstash serverless Redis cluster as key-value pairs (short link -> long link).
  • Every time you visit a short link, it triggers an event and stores it in Upstash Kafka.
  • We then get the data from Upstash Kafka and analyze it in Materialize in real-time.

Diagram

The following is a diagram of the app structure:

Demo

Here is a quick demo of how the app works:

Prerequisites

Before you get started, you need to make sure that you have the following

  • A Redis cluster and a Kafka cluster in Upstash.
  • A Kafka topic in Upstash called visits-log.
  • The Cloudflare CLI tool called wrangler on your local machine as described here
  • A Materialize instance running on your local machine as described here or a Materialize Cloud instance.

Running this demo

Once you have all the prerequisites, you can proceed with the following steps:

  • Clone the repository and run the following command:
git clone https://github.com/bobbyiliev/cf-url-shortener.git
  • Access the directory:
cd cf-url-shortener
  • Install the npm dependencies:
npm install
  • Run the wrangler command to authenticate with Cloudflare:
wrangler login
  • Then in the wrangler.toml file, update the account_id to match your Cloudflare account ID:
account_id = "YOUR_ACCOUNT_ID_HERE"
  • Set the following secrets in Cloudflare using the wrangler tool:
wrangler secret put UPSTASH_REDIS_REST_URL
wrangler secret put UPSTASH_REDIS_REST_TOKEN
wrangler secret put UPSTASH_KAFKA_REST_URL
wrangler secret put UPSTASH_KAFKA_REST_USERNAME
wrangler secret put UPSTASH_KAFKA_REST_PASSWORD
  • Run the following command to deploy the CF Worker:
wrangler deploy

Setup Materialize

Once you’ve deployed the CF Worker, you can set up Materialize to analyze the data in Upstash Kafka in real-time.

Create a Kafka Source

The CREATE SOURCE statements allow you to connect Materialize to an external Kafka data source and lets you interact with its data as if the data were in a SQL table.

CREATE SOURCE click_stats
FROM KAFKA BROKER 'UPSTASH_KAFKA_BROKER_URL' TOPIC 'visits-log'
WITH (
security_protocol = 'SASL_SSL',
sasl_mechanisms = 'SCRAM-SHA-256',
sasl_username = 'UPSTASH_KAFKA_BROKER_USERNAME',
sasl_password = 'UPSTASH_KAFKA_BROKER_PASSWORD'
)
FORMAT BYTES;
CREATE VIEW click_stats_v AS
SELECT
*
FROM (
SELECT
(data->>'shortCode')::string AS short_code,
(data->>'longUrl')::string AS long_url,
(data->>'country')::string AS country,
(data->>'city')::string AS city,
(data->>'ip')::string AS ip
FROM (
SELECT CAST(data AS jsonb) AS data
FROM (
SELECT convert_from(data, 'utf8') AS data
FROM click_stats
)
)
);
CREATE MATERIALIZED VIEW click_stats_m AS
SELECT
*
FROM click_stats_v;
SELECT * FROM click_stats_m;
CREATE MATERIALIZED VIEW order_by_clicks AS
SELECT
short_code,
COUNT(*) AS clicks
FROM click_stats_m
GROUP BY short_code;
COPY ( TAIL ( SELECT * FROM order_by_clicks ) ) TO STDOUT;

Display the results in Metabase

As Materialize is Postgres-wire compatible, you can use BI tools like Metabase to create business intelligence dashboards using the real-time data streams in your Materialize instance.

Conclusion

Using Materialize to analyze the data in your Upstash Kafka serverless instance is a great way to get real-time insights into your data.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Bobby Iliev

Bobby Iliev

I am a professional System Administrator with a demonstrated history of working in the internet industry. I am a Linux lover