thinkinglanguage

ThinkingLanguage

CI

A purpose-built language for Data Engineering & AI — Modern Problems need Modern Solutions.

ThinkingLanguage (TL) replaces the fragile Python + SQL + YAML + Spark glue-code stack with a single, coherent language where data pipelines, transformations, AI model training, and real-time streaming are first-class language constructs.

Highlights

Installation

Quick install (macOS & Linux)

curl -sSf https://raw.githubusercontent.com/mplusm/thinkinglanguage/main/scripts/install.sh | sh

npx (Node.js)

# Run directly without installing
npx thinkinglanguage run hello.tl

# Or install globally
npm install -g thinkinglanguage

Pre-built binaries

Download the latest release from GitHub Releases.

Linux (x86_64):

curl -LO https://github.com/mplusm/thinkinglanguage/releases/latest/download/tl-x86_64-unknown-linux-gnu.tar.gz
tar xzf tl-x86_64-unknown-linux-gnu.tar.gz
sudo mv tl /usr/local/bin/

macOS (Apple Silicon):

curl -LO https://github.com/mplusm/thinkinglanguage/releases/latest/download/tl-aarch64-apple-darwin.tar.gz
tar xzf tl-aarch64-apple-darwin.tar.gz
sudo mv tl /usr/local/bin/

Windows: Download tl-x86_64-pc-windows-msvc.zip from the releases page, extract, and add tl.exe to your PATH.

From crates.io

cargo install thinkinglanguage
# With SQLite and MCP support
cargo install thinkinglanguage --features "sqlite,mcp"

From source

git clone https://github.com/mplusm/thinkinglanguage.git
cd thinkinglanguage
cargo build --release --features "sqlite,mcp"

Quick Start

# Run a script
tl run hello.tl

# Start the REPL
tl shell

A Taste of TL

Variables and Functions

let name = "world"
let nums = [1, 2, 3, 4, 5]

fn greet(who: string) -> string {
    "Hello, {who}!"
}

let doubled = nums |> map((x) => x * 2) |> filter((x) => x > 4)
let total   = reduce(nums, 0, (acc, x) => acc + x)  // 15
print(greet(name))
print(doubled)
print(floor(3.7))    // 3.0
print(7 % 3)         // 1
print(min(4, 9))     // 4

Data Pipelines

let users = read_csv("users.csv")

// Display results
users
    |> filter(age > 30)
    |> select(name, age, department)
    |> aggregate(by: department, count: count(), avg_age: avg(age))
    |> sort("count", "desc")
    |> show()

// Access rows as List<Map> for programmatic use
let rows = users |> filter(age > 30) |> to_rows()
for row in rows {
    print(row["name"] + " — dept: " + row["department"])
}

AI / Machine Learning

let data = read_csv("iris.csv")
let model = train_model(data, target: "species", algorithm: "random_forest")
let predictions = predict(model, new_data)

let t = tensor_ones([4, 4])
let result = t |> matmul(t)

AI Agents

fn search(query) { http_request("GET", "https://api.example.com/search?q=" + query, none, none).body }

agent research_bot {
    model: "gpt-4o",
    system: "You are a research assistant. Use tools to find information.",
    tools {
        search: {
            description: "Search the web",
            parameters: { type: "object", properties: { query: { type: "string" } }, required: ["query"] }
        }
    },
    max_turns: 5
}

let result = run_agent(research_bot, "What is the capital of France?")
println(result.response)

MCP Integration (Model Context Protocol)

Connect to external tool servers or expose TL functions as an MCP server. Requires building with --features mcp.

// Connect to an MCP server and use its tools
let client = mcp_connect("npx", ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"])

let tools = mcp_list_tools(client)
for tool in tools {
    println(tool["name"] + " - " + tool["description"])
}

let result = mcp_call_tool(client, "read_file", {"path": "/tmp/data.txt"})
println(result)

mcp_disconnect(client)

// Use MCP servers with AI agents — tools are auto-discovered
let fs_server = mcp_connect("npx", ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"])

agent file_bot {
    model: "gpt-4o",
    system: "You help users manage files. Use your tools to read and list files.",
    mcp_servers: [fs_server],
    max_turns: 5
}

let result = run_agent(file_bot, "List all .csv files in /tmp")
println(result.response)
mcp_disconnect(fs_server)

See the full MCP guide for server mode, resources, prompts, and more.

Streaming

pipeline etl_daily {
    schedule: "0 6 * * *"
    steps {
        extract read_csv("raw_data.csv")
        transform |> filter(status == "active") |> select(id, name, score)
        load write_parquet("clean_data.parquet")
    }
}

Architecture

Source (.tl)
    │
    ▼
┌─────────┐    ┌──────────┐    ┌─────────┐    ┌───────────┐
│  Lexer  │───▶│  Parser  │───▶│   AST   │───▶│ Compiler  │
│ (logos) │    │(rec.desc)│    │         │    │           │
└─────────┘    └──────────┘    └─────────┘    └─────┬─────┘
                                                    │
                              ┌──────────────────┬──┴──────────────┐
                              ▼                  ▼                 ▼
                        ┌──────────┐      ┌───────────┐    ┌────────────┐
                        │ Bytecode │      │   TL-IR   │    │    LLVM    │
                        │    VM    │      │ Optimizer  │    │  Backend   │
                        │(default) │      │           │    │  (AOT)     │
                        └──────────┘      └───────────┘    └────────────┘
                              │                                   │
                              ▼                                   ▼
                        ┌──────────┐                       ┌────────────┐
                        │   WASM   │                       │   Native   │
                        │ Backend  │                       │  Binary    │
                        └──────────┘                       └────────────┘

Feature Flags

TL uses Cargo feature flags for optional integrations:

Flag Description Dependencies
sqlite SQLite connector rusqlite (bundled)
duckdb DuckDB connector (Arrow-native) duckdb (bundled)
iceberg Apache Iceberg tables (Arrow-native) iceberg
mysql MySQL connector mysql client libs
mssql SQL Server / MSSQL connector tiberius
redis Redis connector redis
s3 S3 object storage object_store
snowflake Snowflake data warehouse reqwest
bigquery Google BigQuery reqwest
databricks Databricks SQL reqwest
clickhouse ClickHouse analytics DB reqwest
mongodb MongoDB connector mongodb driver
sftp SFTP/SCP file transfer ssh2 (libssh2)
kafka Kafka streaming rdkafka
python Python FFI bridge pyo3
gpu GPU tensor operations wgpu, bytemuck
llvm-backend LLVM AOT compilation inkwell, llvm-sys
async-runtime Tokio async I/O tokio, reqwest
mcp Model Context Protocol client/server rmcp, tokio, axum
notebook Interactive notebook TUI ratatui, crossterm
registry Package registry client reqwest

PostgreSQL and Redshift connectors are always available (no feature flag needed).

# Build with specific features
cargo build --release --features "sqlite,gpu,async-runtime"

# Build with all database connectors + SFTP
cargo build --release --features "sqlite,duckdb,iceberg,mysql,mssql,clickhouse,snowflake,bigquery,databricks,mongodb,redis,sftp"

# Build with all features (except llvm-backend which needs LLVM 19 installed)
cargo build --release --features "sqlite,duckdb,iceberg,mysql,mssql,redis,s3,sftp,snowflake,bigquery,databricks,clickhouse,mongodb,kafka,python,gpu,async-runtime,notebook,registry"

CLI Commands

Command Description
tl run <file> Execute a .tl source file
tl shell Start the interactive REPL
tl check <file> Type-check without executing
tl test <path> Run tests in a file or directory
tl fmt <path> Format source files
tl lint <path> Lint for style and correctness
tl build Build the current project (requires tl.toml)
tl init <name> Initialize a new TL project
tl doc <path> Generate documentation
tl disasm <file> Disassemble bytecode
tl compile <file> Compile to native object file (LLVM)
tl deploy <file> Generate deployment artifacts
tl lineage <file> Show data lineage
tl notebook <file> Open interactive notebook
tl lsp Start the LSP server
tl models list\|info\|delete Manage model registry
tl migrate apply\|check\|diff Schema migration
tl add <pkg> Add a dependency
tl remove <pkg> Remove a dependency
tl install Install all dependencies
tl update [pkg] Update dependencies (with version diffs)
tl update --dry-run Preview dependency updates without changes
tl outdated Show outdated dependencies
tl publish Publish to package registry
tl search <query> Search package registry

Project Structure

crates/
  tl-lexer/          Tokenization (logos)
  tl-ast/            Abstract syntax tree
  tl-parser/         Recursive descent parser
  tl-interpreter/    Tree-walking interpreter
  tl-compiler/       Bytecode compiler + register VM
  tl-types/          Type system and checker
  tl-errors/         Error types and diagnostics
  tl-data/           Data engine (DataFusion, Arrow)
  tl-ai/             AI/ML (ndarray, linfa, ONNX, LLM, agents)
  tl-stream/         Streaming and pipelines
  tl-ir/             Intermediate representation optimizer
  tl-llvm/           LLVM AOT backend (inkwell)
  tl-wasm/           WASM browser backend
  tl-gpu/            GPU tensor operations (wgpu)
  tl-lsp/            Language Server Protocol
  tl-package/        Package manager
  tl-mcp/            MCP client/server (rmcp)
  tl-registry/       Package registry server
  tl-cli/            CLI entry point
editors/
  vscode/            VS Code extension
benchmarks/          Criterion benchmarks
examples/            Example .tl programs
docs/                Documentation

Documentation

Detailed documentation is available in the docs/ directory:

Tests

# Run all tests (excluding feature-gated crates)
cargo test --workspace --exclude tl-llvm --exclude tl-wasm --exclude tl-gpu

# With specific features
cargo test --workspace --exclude tl-llvm --exclude tl-wasm --exclude tl-gpu --features sqlite

License

Apache-2.0

This site is open source. Improve this page.