My Awesome Data Engineer

1. Databases

• Bytebase - Database devops - https://www.bytebase.com/
• DuckDB - SQL OLAP database management system - https://duckdb.org/
• Harlequin.sh - A drop-in replacement for the DuckDB CLI. - https://harlequin.sh/

2. Python

• RustPython (python intepreter written in Rust) > GoPython > PythonPy

3. Computation

• SparkMeasure - measure and monitor spark job https://github.com/LucaCanali/sparkMeasure

4. Kubernetes

• EKS Distro - An open source Kubernetes distribution based on EKS.
• Karpenter - Just-in-time Nodes for Any Kubernetes Cluster - https://karpenter.sh/
• KubeCost - Monitor and Reduce Kubernetes Spend - https://www.kubecost.
• KCP - Kubernetes-like control plane https://github.com/kcp-dev/kcp

5. Visualisation

• BI tool + Markdown https://evidence.dev/blog