fix: get Xiaohongshu fulltext #17075

rien7 · 2024-10-10T14:27:43Z

Involved Issue / 该 PR 相关 Issue

Example for the Proposed Route(s) / 路由地址示例

/xiaohongshu/user/5b55f1534eacab0302da1f02/notes/fulltext

New RSS Route Checklist / 新 RSS 路由检查表

New Route / 新的路由
- Follows Script Standard / 跟随路由规范
Anti-bot or rate limit / 反爬/频率限制
- If yes, do your code reflect this sign? / 如果有, 是否有对应的措施?
Date and time / 日期和时间
- Parsed / 可以解析
- Correct time zone / 时区正确
New package added / 添加了新的包
Puppeteer

Note / 说明

Need to add XIAOHONGSHU_COOKIE

lib/routes/ipswdev/index.ts

lib/routes/xiaohongshu/notes.ts

github-actions · 2024-10-10T14:33:04Z

Successfully generated as following:

http://localhost:1200/xiaohongshu/user/5b55f1534eacab0302da1f02/notes/fulltext - Failed ❌

HTTPError: Response code 503 (Service Unavailable)

Error Message:<br/>Error: Could not get user information and note list
TargetCloseError: Page closed!
Route: /xiaohongshu/user/:user_id/notes/fulltext
Full Route: /xiaohongshu/user/5b55f1534eacab0302da1f02/notes/fulltext
Node Version: v22.9.0
Git Hash: 7c52019d

github-actions · 2024-10-10T14:55:54Z

Successfully generated as following:

http://localhost:1200/xiaohongshu/user/5b55f1534eacab0302da1f02/notes/fulltext - Failed ❌

HTTPError: Response code 503 (Service Unavailable)

Error Message:<br/>Error: Could not get user information and note list
TargetCloseError: Page closed!
Route: /xiaohongshu/user/:user_id/notes/:fulltext
Full Route: /xiaohongshu/user/5b55f1534eacab0302da1f02/notes/fulltext
Node Version: v22.9.0
Git Hash: ccb76194

github-actions · 2024-10-10T15:08:56Z

Successfully generated as following:

http://localhost:1200/xiaohongshu/user/5b55f1534eacab0302da1f02/notes/fulltext - Failed ❌

HTTPError: Response code 503 (Service Unavailable)

Error Message:<br/>Error: Could not get user information and note list
TargetCloseError: Page closed!
Route: /xiaohongshu/user/:user_id/notes/:fulltext
Full Route: /xiaohongshu/user/5b55f1534eacab0302da1f02/notes/fulltext
Node Version: v22.9.0
Git Hash: 06343cfa

github-actions · 2024-10-11T05:17:43Z

Successfully generated as following:

http://localhost:1200/xiaohongshu/user/5b55f1534eacab0302da1f02/notes/fulltext - Failed ❌

HTTPError: Response code 503 (Service Unavailable)

Error Message:<br/>Error: Could not get user information and note list
TargetCloseError: Page closed!
Route: /xiaohongshu/user/:user_id/notes/:fulltext
Full Route: /xiaohongshu/user/5b55f1534eacab0302da1f02/notes/fulltext
Node Version: v22.9.0
Git Hash: 7ad18389

rien7 · 2024-10-14T03:01:14Z

有希望吗

需要自建

zucchiniEvader · 2024-10-14T03:03:05Z

有希望吗

需要自建

好的，我拉下你的代码试下

github-actions · 2024-10-14T07:04:10Z

Successfully generated as following:

http://localhost:1200/xiaohongshu/user/5b55f1534eacab0302da1f02/notes/fulltext - Failed ❌

HTTPError: Response code 503 (Service Unavailable)

Error Message:<br/>Error: Could not get user information and note list
TargetCloseError: Page closed!
Route: /xiaohongshu/user/:user_id/notes/:fulltext
Full Route: /xiaohongshu/user/5b55f1534eacab0302da1f02/notes/fulltext
Node Version: v22.9.0
Git Hash: 7e168ae9

dddaniel1 · 2024-10-20T06:42:19Z

有希望吗

需要自建

代码似乎还有点问题，我拉你仓库改了下，能用了。

zucchiniEvader · 2024-10-23T06:20:19Z

会合并吗 @DIYgod

TonyRL · 2024-11-19T12:17:47Z

lib/routes/xiaohongshu/notes.ts

+async function getUser(url, cookie) {
+    const res = await got(url, {
+        headers: {
+            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36',


RSSHub will use a randomised user agent of Chrome on mac by default. Does the site only work with this fixed version of Chrome on Windows?

TonyRL · 2024-11-19T12:17:51Z

lib/routes/xiaohongshu/notes.ts

+    const data = (await cache.tryGet(link, async () => {
+        const res = await got(link, {
+            headers: {
+                'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36',


RSSHub will use a randomised user agent of Chrome on mac by default. Does the site only works with this fixed version of Chrome on Windows?

TonyRL · 2024-11-19T12:26:41Z

lib/routes/xiaohongshu/notes.ts

+    const res = await got(url, {
+        headers: {
+            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36',
+            Cookie: cookie,
+        },
+    });


This request does not require cookie.

TonyRL · 2024-11-19T12:36:41Z

lib/routes/xiaohongshu/notes.ts

+    let script = $('script')
+        .filter((i, script) => {
+            const text = script.children[0]?.data;
+            return text?.startsWith('window.__INITIAL_STATE__=');
+        })
+        .text();
+    script = script.slice('window.__INITIAL_STATE__='.length);
+    script = script.replaceAll('undefined', 'null');


Suggested change

let script = $('script')

.filter((i, script) => {

const text = script.children[0]?.data;

return text?.startsWith('window.__INITIAL_STATE__=');

})

.text();

script = script.slice('window.__INITIAL_STATE__='.length);

script = script.replaceAll('undefined', 'null');

const script = $("script:contains('__INITIAL_STATE__')")

.text()

.match(/window\.__INITIAL_STATE__=(.*)/)?.[1]

?.replaceAll('undefined', 'null');

TonyRL · 2024-11-19T12:39:17Z

lib/routes/xiaohongshu/notes.ts

+async function renderNotesFulltext(notes, url) {
+    const data: any[] = [];
+    const promises = notes.flatMap((note) =>
+        note.map(async ({ noteCard }) => {
+            const link = `${url}/${noteCard.noteId}`;
+            const { title, description, pubDate } = await getFullNote(link);
+            return {
+                title,
+                link,
+                description,
+                author: noteCard.user.nickName,
+                guid: noteCard.noteId,
+                pubDate,
+            };
+        })
+    );
+    data.push(...(await Promise.all(promises)));
+    return data;
+}


Please remove unnecessary spreading and push

Suggested change

async function renderNotesFulltext(notes, url) {

const data: any[] = [];

const promises = notes.flatMap((note) =>

note.map(async ({ noteCard }) => {

const link = `${url}/${noteCard.noteId}`;

const { title, description, pubDate } = await getFullNote(link);

return {

title,

link,

description,

author: noteCard.user.nickName,

guid: noteCard.noteId,

pubDate,

};

})

);

data.push(...(await Promise.all(promises)));

return data;

}

function renderNotesFulltext(notes, url) {

const promises = notes.flatMap((note) =>

note.map(async ({ noteCard }) => {

const link = `${url}/${noteCard.noteId}`;

const { title, description, pubDate } = await getFullNote(link);

return {

title,

link,

description,

author: noteCard.user.nickName,

guid: noteCard.noteId,

pubDate,

};

})

);

return Promise.all(promises);

}

TonyRL · 2024-11-19T12:39:29Z

lib/routes/xiaohongshu/notes.ts

+        let script = $('script')
+            .filter((i, script) => {
+                const text = script.children[0]?.data;
+                return text?.startsWith('window.__INITIAL_STATE__=');
+            })
+            .text();
+        script = script.slice('window.__INITIAL_STATE__='.length);
+        script = script.replaceAll('undefined', 'null');


same as above

github-actions · 2024-11-21T13:43:05Z

Successfully generated as following:

http://localhost:1200/xiaohongshu/user/5b55f1534eacab0302da1f02/notes/fulltext - Failed ❌

HTTPError: Response code 503 (Service Unavailable)

Error Message:<br/>Error: Could not get user information and note list
TargetCloseError: Page closed!
Route: /xiaohongshu/user/:user_id/notes/:fulltext
Full Route: /xiaohongshu/user/5b55f1534eacab0302da1f02/notes/fulltext
Node Version: v22.11.0
Git Hash: dca14f95

rien7 added 6 commits September 30, 2024 23:47

[feat(routes)] Add ipsw.dev support

f99b215

Merge branch 'DIYgod:master' into master

416603a

feat: add fulltext with cookie

a5ee720

feat: add fulltext with cookie

25ba2e4

Merge branch 'refs/heads/master' into xiaohongshu

fe9df5c

Merge branch 'DIYgod:master' into xiaohongshu

c79a5e1

github-actions bot added the Route label Oct 10, 2024

github-advanced-security bot found potential problems Oct 10, 2024

View reviewed changes

lib/routes/ipswdev/index.ts Fixed Show fixed Hide fixed

lib/routes/xiaohongshu/notes.ts Fixed Show fixed Hide fixed

github-actions bot added the Auto: Route Test Complete Auto route test has finished on given PR label Oct 10, 2024

fix: config

3434cd4

fix: await in loop

5399b18

fix: remove ipswdev in other branch

650c858

This comment was marked as off-topic.

Sign in to view

fix: add pubDate

09cfcee

pseudoyu force-pushed the master branch 5 times, most recently from 8ef9ae7 to d224d67 Compare October 31, 2024 11:39

TonyRL force-pushed the master branch from 4c64301 to 0ccfd86 Compare November 6, 2024 21:07

TonyRL reviewed Nov 19, 2024

View reviewed changes

Merge branch 'master' into xiaohongshu

0f2f5cc

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: get Xiaohongshu fulltext #17075

fix: get Xiaohongshu fulltext #17075

rien7 commented Oct 10, 2024

github-actions bot commented Oct 10, 2024

github-actions bot commented Oct 10, 2024

github-actions bot commented Oct 10, 2024

github-actions bot commented Oct 11, 2024

This comment was marked as off-topic.

rien7 commented Oct 14, 2024

zucchiniEvader commented Oct 14, 2024

github-actions bot commented Oct 14, 2024

dddaniel1 commented Oct 20, 2024

zucchiniEvader commented Oct 23, 2024

TonyRL Nov 19, 2024

TonyRL Nov 19, 2024

TonyRL Nov 19, 2024

TonyRL Nov 19, 2024

TonyRL Nov 19, 2024

TonyRL Nov 19, 2024

github-actions bot commented Nov 21, 2024

fix: get Xiaohongshu fulltext #17075

Are you sure you want to change the base?

fix: get Xiaohongshu fulltext #17075

Conversation

rien7 commented Oct 10, 2024

Involved Issue / 该 PR 相关 Issue

Example for the Proposed Route(s) / 路由地址示例

New RSS Route Checklist / 新 RSS 路由检查表

Note / 说明

github-actions bot commented Oct 10, 2024

github-actions bot commented Oct 10, 2024

github-actions bot commented Oct 10, 2024

github-actions bot commented Oct 11, 2024

This comment was marked as off-topic.

rien7 commented Oct 14, 2024

zucchiniEvader commented Oct 14, 2024

github-actions bot commented Oct 14, 2024

dddaniel1 commented Oct 20, 2024

zucchiniEvader commented Oct 23, 2024

TonyRL Nov 19, 2024

Choose a reason for hiding this comment

TonyRL Nov 19, 2024

Choose a reason for hiding this comment

TonyRL Nov 19, 2024

Choose a reason for hiding this comment

TonyRL Nov 19, 2024

Choose a reason for hiding this comment

TonyRL Nov 19, 2024

Choose a reason for hiding this comment

TonyRL Nov 19, 2024

Choose a reason for hiding this comment

github-actions bot commented Nov 21, 2024