Coder Social home page Coder Social logo

yukaii / zuzugo Goto Github PK

View Code? Open in Web Editor NEW
9.0 5.0 3.0 1000 KB

租租狗,通通給我租起來!

Home Page: https://zuzugo.vercel.app

License: MIT License

CSS 11.84% TypeScript 86.68% JavaScript 1.48%
591 bot cralwer line-notify nextjs slack-bot typescript inngest

zuzugo's Introduction

Zuzugo 租租狗

logo

抓取 591 最新租屋資訊,並提供聊天機器人通知(可簡易的擴充)

Status page: https://zuzugo.instatus.com/

Screenshots

Line Notify Slack Bot
line-notify slack-bot

Roadmap

參見 GitHub Project

Development

  1. Clone 專案,然後
pnpm install
cp .env.example .env

# start the inngest dev server
pnpm inngest-dev

Previous Works

Misc:關於命名

  • 豬豬快租 -> zuzu.com.tw
  • bbgo -> BB 狗,雖然本專案沒有半行 Go...

yeah

Slack setup

  1. Go to https://zuzugo.tunnelto.dev/api/slack/install
  2. Click Add to Slack channel, select the channel you want to receive the notification
  3. Check the database, you should see SlackAppInstallation table has a new record

License

MIT

zuzugo's People

Contributors

alan10332000 avatar moojing avatar steward379 avatar yukaii avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

zuzugo's Issues

Refactor slack config

zuzugo/lib/config.ts

Lines 40 to 49 in a2222fa

slackWebhook,
slackAppToken: process.env.SLACK_APP_TOKEN,
slackBotToken: process.env.SLACK_BOT_TOKEN,
slackSigningSecret: process.env.SLACK_SIGNING_SECRET,
slackDevMode: process.env.SLACK_DEV_MODE === "true",
slackClientId: process.env.SLACK_CLIENT_ID,
slackClientSecret: process.env.SLACK_CLIENT_SECRET,
slackStateSecret: process.env.SLACK_STATE_SECRET,
slackRedirectUri: process.env.SLACK_REDIRECT_URI,
slackSlashCommand: process.env.SLACK_SLASH_COMMAND,

Refactor to scoped config like:

slack: {
  incomingWebhook: process.env.INCOMING_WEBHOOK,
  ...
}

Linebot intergration

Now we have line notification for notifying renting data, but we could have more flexibility if we integrated line bot API.

Things to do

  • Add line web hook for getting userId or roomId. (ref: linebot SDK and webhook api)
  • Notify renting data using line bot API.
  • Store groupId or userId into the database. ( waiting for finishing the migration to Prisma that's currently doing by @Yukaii )
  • integrate line login channel ( waiting for finishing the migration to Prisma that's currently doing by @Yukaii )

New house cleanup mechanism

目前是按照時間排序,超過 100 筆就砍了

之後可以改成按照時序排序,超過兩週就砍

等到 #23 有 deliver 的話,有 star 過的可以延長到一個月

未來如果爬的資料不止租屋,而是售屋,那可以再延長(甚至是 💰 的部分)

Randomize User-Agent header

zuzugo/lib/api.ts

Lines 12 to 15 in e86554f

headers: {
"User-Agent":
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36",
},

RSSHub 是有個隨機值的,可以參考下

Adopt inngest

https://vercel.com/integrations/inngest

用這個就沒有 response time 的限制了(應該吧),可以把各個 notification module 拆散到 function 去

本來想說把後端拆出去,但維持 monolith & simple 才是本專案好維護的方式,不要讓 infra 也模組化啦!這只是個超級好一鍵部屬的 Side Project,知道了!

Run build command in CI

現在只有跑 lint 和 test 而已,需要增加 build (使用 pnpm run build command)

test:
name: Test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: ./.github/actions/setup-repo
- name: Run tests
run: pnpm test

臉書社團資料來源

用簡易網頁版來爬最新資料就好

  • ex: https://mbasic.facebook.com/groups/464870710346711

轉成 RSS Hub 格式,讓大家自己去更新

54th g0v 大松後狀態更新

@yellowsoar 討論後發現,單純以訂閱功能做考量,用 RSSHub 更適合,不但不需要考慮:

  1. 通知平臺,Any tools
  2. 資料快取問題
  3. 資料更新問題

以一個熱門物件來說,比起優先實作追蹤特定物件及追蹤價格,從「多個管道」、獲取資訊還比較重要,好的物件也不只有 591 有(ex: 臉書社團);如果需要串接通知到其他即時通訊平臺 (slack/line/discord/telegram),可以用 n8n/zapier 之類的自動化工具就好。

雖然 slack 的 Block UI 實在很有趣,但對於通知來說確實只是自我滿足 XD 點進去看就好

結論

Free tier 就用 RSS 就好,更高頻率的更新可以自己 host,或是請給我錢

目前 591 的 RSS Source 已更新在 PR

聲明 & 法規研究

聲明

本軟體專案(以下簡稱「本專案」),目的在於教育及展示 React.js、Next.js、Vercel 平臺等,相關技術的運用及架構。為遵守相關法律法規,我們在此聲明如下:

  1. 本專案不會以公開方式架設的方式提供一般消費者任何服務,亦不會收取費用。
  2. 本專案所包含的爬蟲程式碼僅作為教育及演示需求,爬蟲所得到的資料僅為自用,不會用於任何商業或營利目的。本專案尊重著作權,並承諾不會以任何形式侵害他人的智慧財產權或其他權益。
  3. 本專案的爬蟲程式碼可能涉及到公平交易法的相關問題,我們將盡最大努力遵守相關法律法規,避免任何侵權或違法行為。若因使用本專案而造成任何損害或違法行為,本專案概不負責。
  4. 我們建議使用本專案的使用者不要自行架設或使用本專案的爬蟲程式碼,以避免可能的法律風險。如使用本專案的爬蟲程式碼,使用者應留意相關法律法規的規範,確保自己的使用行為符合法律法規。

最後,本專案將不斷遵守相關法律法規,並致力於推動技術發展與社會進步。

Drafted by ChatGPT
CTA 是不是很讚

Multi-user notification with HouseNotification model

About the new house crawling mechanism

The house store stores the most recent data. There are two types of cron jobs to perform crawling:

  • Main Job: Crawl the default raw query (including all new houses) every 3 minutes.
  • Subscription Jobs: Extract and create unique subscription queries for each channel, execute them independently, and then send out notifications. The frequency of execution may vary according to the user's plan.

New notification system

Notification is sent per channel. So we need new models for sending notification.

  • Subscription <-- hasMany --> ChannelNotification <-- hasMany --> House
erDiagram
  Subscription ||--o{ ChannelNotification : hasMany
  House ||--o{ ChannelNotification : hasMany

Slack commands

寫更多互動的 Slack 功能,比如:建立關注清單、價格變更、是否已出租

phelia (沒在維護了,用 bolt.js 就好)

V1 功能規劃

一樣拆成幾個階段來做吧:

  • #11 這個撿回來先做
  • #88
  • #92
  • #94
  • #96
    • hasMany through relation config for prisma
  • #100
  • #101
  • #102
  • UserPreference model and Line subscription (manually)
  • TBD: Star

Models

https://beta.tldraw.com/r/v2_c_w1x0VHOIFp6gvD1yXjv17

Image

#42 之後,一些比較進階需要關聯試資料庫的功能也比較容易實作了,大致整理成以上表格。

比較未知的算是 Slack 整合,想做到如同 GitHub Slack App 一樣的體驗,能在各個 Channel 呼叫 bot command 來訂閱通知到該 channel。(阿結果 GitHub Slack App 也沒有開源)為此需要多開表格。其他部分就相對容易,CRUD 工程師嘛。

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.