Skip to content
View myps6415's full-sized avatar

Block or report myps6415

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. Markdown is supported. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
myps6415/README.md

πŸ‘‹ Hi, I'm John Tung (@myps6415)

Principal Data Engineer Β· Modern Data Stack on GCP Β· OSS Contributor

⭐ Follow me on GitHub β†’

🌐 English Β· δΈ­ζ–‡

πŸ“„ View my CV β†’


πŸš€ About Me

  • 🌏 Based in Taiwan, working as a Principal Data Engineer at a media group
  • πŸ› οΈ Building the social analytics data platform from the ground up β€” ingestion, modeling, dashboards, deployment, docs
  • 🧰 Production work across GCP, AWS, and Azure; orchestration with Airflow; data processing with PySpark β€” current GCP-lean stack is chosen by fit, not by habit
  • πŸ‘₯ Previously Data Team Lead for 2.5 years (Scrum, mentoring, hiring, vendor coordination) before stepping back from management to a Principal IC role β€” always kept building hands-on throughout
  • πŸ”¬ Came up through text mining, NLP, and sentiment analysis on Chinese-language corpora β€” direct context for the current LLM-augmented pipeline work
  • 🎯 Bias toward lean, observable, cost-disciplined data stacks (BigQuery + Cloud Run Jobs + dbt) over heavy orchestrators
  • πŸ” Spend equal time on data modeling and on the boring infra that keeps tokens fresh, jobs idempotent, and bills small
  • 🏠 Run a homelab with the same discipline as the day job β€” a MacBook server hosting Docker stacks (photo backup, monitoring, a family-facing LINE AI assistant, a self-hosted agent runtime), each wrapped in its own watchdog loops
  • πŸ“š Currently going deep on LLM-augmented data pipelines (sentiment / topic classification) and vector search for editorial workflows

πŸ› οΈ Technologies & Tools

Data Engineering BigQuery dbt Cloud Run Airflow PySpark Apache Iceberg Fluentd

Cloud GCP AWS Azure Kubernetes Cloudflare Pages

AI / GenAI / LLMOps Gemini scikit-learn LightGBM NLP

Data Analysis & Viz Streamlit Looker Studio Superset Plotly pandas

DevOps / CI-CD Docker Cloud Build GitHub Actions Ansible Poetry pytest Ruff

Languages & Database Python SQL MySQL MongoDB Redis Elasticsearch

Working with β€” patterns and tools I reach for
  • Ingestion: Cloud Run Jobs + Cloud Scheduler (chosen over Airflow / Composer for cost), Cloud Build CI/CD
  • Modeling: dbt-bigquery β€” append-only raw ingestion, then dbt staging (views) / marts (tables) with QUALIFY ROW_NUMBER() dedup
  • APIs: Meta Graph API (Facebook / Instagram), Threads API (OAuth 2.0), YouTube Data API v3
  • Secrets & auth: GCP Secret Manager, OAuth 2.0 long-lived token refresh automation
  • LLM-augmented pipelines: Gemini via BigQuery AI.GENERATE for classification + grounded generation (shipped), Vertex embeddings + KMEANS clustering + RAG retrieval with LLM-as-judge eval gating (shipped), BigQuery Vector Search for the editorial archive (in flight)
  • Past production: GKE-hosted Fluentd telemetry at high QPS, Apache Iceberg + Spark data lake, multi-cloud (AWS ↔ GCP) ETL with Ansible-driven deployment, classical ML (XGBoost / RandomForest / SVM) for user-segment prediction
  • Other: Poetry, pytest, ruff (lint + format in CI), GitHub Actions, SendGrid / SMTP email-reporting framework

πŸ’Ό Recent Work

Shipped

  • Multi-platform social ingestion β€” end-to-end pipelines for Facebook, Instagram, Threads, and YouTube into BigQuery (15+ Cloud Run Jobs on staggered Cloud Scheduler triggers, append-only + dedup-on-read for full history), with GA4 site/app traffic added on top. Powers the dashboards editorial and marketing teams open every morning. A data-quality postmortem from this pipeline: when a missing value must be NULL, not 0 β†’
  • Two production operating dashboards β€” built two Streamlit dashboards on Cloud Run with Google OIDC login (one for group-wide digital revenue, one for social-media performance). A self-serve permission system (3-tier roles + change audit) lets managers add/remove members themselves without a redeploy, and event tracking feeds product- and feature-adoption analysis. On-dashboard AI insights share a persistent cache so cost stays flat regardless of users
  • Threads OAuth 2.0, end-to-end β€” dual-account 8-scope authorization, plus a weekly Cloud Run Job that auto-refreshes the 60-day token through Secret Manager. Upgraded the original "alert + manual re-auth" design into a fully unattended one
  • Self-serve OAuth UX for non-engineers β€” pure-frontend authorization helper on Cloudflare Pages so social-team editors can grant API access without engineering hand-holding
  • Inherited pipeline rewrite β€” took over a Composer + Apps Script revenue pipeline; replaced it with BigQuery External Tables + Scheduled Queries, cutting monthly cost from ~USD$300 to under $1 with no loss of functionality
  • Orchestration decision β€” explicitly skipped managed Airflow (Composer) on cost grounds; chose Cloud Scheduler + Cloud Run Jobs for the current scale, with a documented migration path to a dedicated orchestrator when the workload justifies it
  • Silent failure detection β€” dbt source freshness (25h warn / 49h error) + Cloud Monitoring alerts on regex-matched job failures, so a broken upstream API or crashed job can't go undetected for days
  • Personal automation self-hosting (June 2026) β€” Migrated a Zeabur-hosted n8n workflow (daily family-schedule alert pushed to LINE) to a 178-line stdlib Python script triggered by local OpenClaw cron, dropping the $7.88/month bill to $0 once OpenClaw had been running long enough at home to make the hosted workflow redundant. Fixed a latent all-day-event crash bug as a side effect of the rewrite. Full migration writeup β†’
  • LINE family AI assistant + six-loop ops system (July 2026) β€” Self-hosted LINE group bot on the home server (FastAPI webhook behind a Cloudflare Tunnel): 14 function-calling tools β€” weather, quotes, FX, recurring reminders that @-mention people, price alerts gated by market hours + quote freshness (no holiday calendar to maintain), chat recap, vision, long-term memory β€” presented as LINE Flex cards with one-tap follow-ups, all on the free reply API and a flat-rate LLM so marginal cost per message is $0. Wrapped in six independent watchdog loops running outside Docker (liveness, daily health digest, delivery audit that catches promised-but-never-sent reminders, verified weekly backups, daily regression probes with deterministic assertions), because a webhook bot's worst failure mode is silence. Series writeup β†’
  • This site β†’ MDX content collections (June 2026) β€” Migrated the blog from per-page Astro to a bilingual MDX content collection and added a self-hosted Sveltia CMS (Cloudflare Worker OAuth, $0) for browser editing. Caught two silent bugs en route β€” a reserved field name that nondeterministically dropped posts, and a CommonMark rule that won't bold CJK text. Full writeup β†’
  • LLM-augmented data pipelines β€” Gemini 2.5 Flash (via BigQuery AI.GENERATE) for comment sentiment and post topic classification, surfaced as fct_comments_sentiment / fct_post_topics marts that dashboards and analysts query directly (incremental β€” only new rows); topic labels match the newsroom's own section taxonomy; an ai_cache keeps on-demand inference cost flat. How I kept the bill from scaling with users β†’
  • Semantic clustering + RAG retrieval foundation β€” Vertex embeddings (ML.GENERATE_EMBEDDING, 768-dim multilingual) + KMEANS cluster negative comments, with an LLM naming each theme ("what are readers complaining about"); a comment RAG retrieval layer is gated by a weekly LLM-as-judge eval (two-tier thresholds; the job fails if quality drops) that guards against code regressions and model/data drift
  • Competitor benchmarking + statistical cross-BU analysis β€” competitor ingestion via official public APIs (YouTube + Instagram) and topic share-of-voice; external factors (stock-market index, national business-cycle signal) joined with Mann-Kendall trend tests, Sen's slope, seasonal decomposition, and lead-lag correlation to separate tailwind growth from genuine against-the-trend wins
  • Revenue-table PDF generator β€” Streamlit + ReportLab parse three TSE/OTC/EMG .xls files into a high-fidelity monthly-revenue newspaper-page PDF, replacing manual layout
  • GA4 web-analytics ingestion β€” the group's site + app traffic pulled in through the GA4 Data API (4 Cloud Run Jobs + an fct_ga_story ranking mart); non-engineers grant Viewer access through a self-serve Cloudflare Pages page

In Flight

  • πŸ”Ž Editorial semantic search β€” the embedding + retrieval foundation is already shipped (negative-comment clustering + comment RAG retrieval); now extending it to the historical article archive, evaluating BigQuery Vector Search vs. self-hosted Elasticsearch so editors can search by meaning
  • 🧹 Migration wrap-up β€” the new architecture is already producing; decommissioning the last revenue pipeline off the old Composer DAG / Apps Script

Side Project

  • realping.tw β€” a Taiwan real-floor-area (ε―¦εͺ) property-transaction database I built solo: dbt-duckdb cleans the government open data, then one shared query core serves it as both a REST API (FastAPI) and an MCP interface (FastMCP), with API keys / usage quotas, self-serve signup, an OAuth dashboard, and a Cloudflare edge guard; a LightGBM valuation model (median error ~12%) is built and held ready for when demand shows up. Live at api / mcp.realping.tw

🌱 Open Source Contributions

  • openclaw/openclaw#87291 (Filed May 2026) β€” Filed a source-level root cause for a silent 500-character truncation in OpenClaw's reply-context sanitizer: one cap covered both short metadata fields and multi-paragraph bot bodies, which clobbered tail content after idle session reset and left Telegram users with "amnesiac" bots and no error trace. Proposed splitting the body cap from the field cap; maintainers later landed a more sophisticated variant of the same split β€” head+tail truncation with additional ReplyChain / inline ReplyToBody coverage β€” in commit 3753c5e2c8. Full postmortem β†’
  • openclaw/openclaw#84890 (Merged May 2026) β€” Root-caused a SIGUSR1-listener / dynamic-import deadlock that left the gateway unable to restart after in-place package upgrades. Shipped a +28 / βˆ’1 surgical fix (eager-load lifecycle runtime + listener .catch) backed by production logs and a live kill -USR1 reproduction on the patched build. Full postmortem β†’

πŸ“ž Connect with Me

LinkedIn Email Medium

Popular repositories Loading

  1. NLP-marathon NLP-marathon Public

    Learn NLP knowledge

    Jupyter Notebook 2

  2. Crawler_Project Crawler_Project Public

    Jupyter Notebook 1 1

  3. LineBot-for-my-mom LineBot-for-my-mom Public archive

    Python

  4. Data-Analysis-Project Data-Analysis-Project Public archive

    Step by step go on data analysis

    Jupyter Notebook

  5. myps6415.github.io myps6415.github.io Public

    MDX

  6. myps6415 myps6415 Public

    Config files for my GitHub profile.