Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: ノートのインポート機能 #21

Closed
wants to merge 59 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
3791394
add: Importing of Posts
Nov 12, 2023
265ed52
upd: update limit
Nov 12, 2023
be307db
upd: pass full thinuser
Nov 12, 2023
f931a9f
fix: cleanup not available on development env
Nov 12, 2023
da91036
upd: remove unused function, fix logger message
Nov 12, 2023
74a378a
fix: broken unicode in strings on IG imports
Nov 12, 2023
234dfaf
fix: use a proper function
Nov 12, 2023
d4e73a9
any→string
kanarikanaru Aug 13, 2024
0a9fcb1
fix: prevent *key renotes/boosts from being imported
Nov 13, 2023
f9e8ea0
upd: import replies to own posts on *key
Nov 13, 2023
8c16ee4
fix: type
Nov 12, 2023
fa959ea
upd: import own replies to own tweets
Nov 17, 2023
26964f2
chore: remove type in DbNoteImportToDbJobData
Nov 13, 2023
d577731
merge: improve recreateChain
Nov 18, 2023
c21e1a9
upd: remove event stream calls on import
Nov 17, 2023
baf0cb8
upd: lower max uses per hour on `import notes`
Nov 30, 2023
1bf1105
upd: add `Facebook` to note importing
Nov 22, 2023
b94e38f
fix: attachments not working on FB import
Nov 22, 2023
0ae7dad
upd: create Folders for imported media
Nov 28, 2023
a151d62
prepare to import more notes
gceccarelli-vt Nov 28, 2023
be1791b
properly thread Mastodon imports
dakkar Nov 30, 2023
e43db98
fix chaining for Mastodon notes
dakkar Nov 30, 2023
ddb04a9
thread Pleroma imports as well
dakkar Nov 30, 2023
5da7658
Use hashed filename for exists check
Dec 31, 2023
a1daa3c
upd: prevent vanilla mastodon imports from importing DMs
Dec 31, 2023
d1e1817
fix: correct followers visibility on import
Dec 31, 2023
7d757c6
fix: if condition
Dec 31, 2023
ee745d7
upd: add home as a visibility for mastodon imports
Dec 31, 2023
94fb2a6
Use hex digest
Dec 31, 2023
a0d6570
Use base64url digest
Dec 31, 2023
7e464ea
Simplify hash steps
Dec 31, 2023
885e700
fix: visibility check on masto import
Dec 31, 2023
3e0a48e
link twitter names to twitter, not nitter #382
dakkar Jan 28, 2024
e9686ae
Fixed code injection from twitter import
ChaoticLeah Feb 1, 2024
306aadb
simpler json-isation
dakkar Feb 3, 2024
184c6fe
simpler mapping
dakkar Feb 3, 2024
74a35b4
make almost all fs ops async
dakkar Feb 3, 2024
15ff18e
simpler logic
dakkar Feb 3, 2024
a2a8c4d
decode entity references from tweets
dakkar Feb 3, 2024
addc348
lint
dakkar Feb 3, 2024
a7fdc35
import fs/promises the right way
dakkar Feb 3, 2024
ccb2f75
fix: keep alt text of file if present
Feb 13, 2024
ea8a034
allow setting separate timeout / max size for imports - fixes #479
dakkar Jun 3, 2024
2c54ac7
Update index.d.ts
kanarikanaru Aug 13, 2024
456a6eb
lint
kanarikanaru Aug 13, 2024
b206179
os.api→misskeyApi
kanarikanaru Aug 13, 2024
b1f79a6
Update QueueProcessorService.ts
kanarikanaru Aug 13, 2024
063cbbe
Chore: remove sharkey specific elements
kanarikanaru Aug 13, 2024
64c4de3
Add check for prohibited words when importing notes
kanarikanaru Aug 13, 2024
289da88
Update ImportNotesProcessorService.ts
kanarikanaru Aug 13, 2024
bbddc04
Fix: DragonflyDB error during note import
kanarikanaru Aug 13, 2024
6c7a9ec
Fix: チャートにダイレクト投稿が反映されないように
kanarikanaru Aug 13, 2024
6dcbc9f
isSensitive→isKeyWordIncluded
kanarikanaru Aug 13, 2024
6e2b870
Enhance: pushしないTLを指定できるように
kanarikanaru Aug 13, 2024
ccdda6d
Chore: Prevent FTT issues from invalid note time
kanarikanaru Aug 14, 2024
1d9f29e
Chore: Add note import info and disclaimer
kanarikanaru Aug 14, 2024
38103c5
Refactor: NoteCreateService
kanarikanaru Aug 14, 2024
efac418
autogen
kanarikanaru Aug 14, 2024
5762dae
Update misskey-js.api.md
kanarikanaru Aug 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
lint
dakkar authored and kanarikanaru committed Aug 13, 2024
commit addc3485cf419c565367c05618a383f474d02513
Original file line number Diff line number Diff line change
@@ -131,7 +131,7 @@ export class ImportNotesProcessorService {
}

@bindThis
private parseTwitterFile(str : string) : null | { tweet: object }[] {
private parseTwitterFile(str : string) : { tweet: object }[] {
const jsonStr = str.replace(/^\s*window\.YTD\.tweets\.part0\s*=\s*/, '');

try {
@@ -191,7 +191,7 @@ export class ImportNotesProcessorService {

const unprocessedTweets = this.parseTwitterFile(await fs.promises.readFile(outputPath + '/data/tweets.js', 'utf-8'));

const tweets = unprocessedTweets.map(e=>e.tweet);
const tweets = unprocessedTweets.map(e => e.tweet);
const processedTweets = await this.recreateChain(['id_str'], ['in_reply_to_status_id_str'], tweets, false);
this.queueService.createImportTweetsToDbJob(job.data.user, processedTweets, null);
} finally {
@@ -584,7 +584,7 @@ export class ImportNotesProcessorService {

try {
const date = new Date(tweet.created_at);
const decodedText = tweet.full_text.replaceAll('&gt;','>').replaceAll('&lt;','<').replaceAll('&amp;','&');
const decodedText = tweet.full_text.replaceAll('&gt;', '>').replaceAll('&lt;', '<').replaceAll('&amp;', '&');
const textReplaceURLs = tweet.entities.urls && tweet.entities.urls.length > 0 ? await replaceTwitterUrls(decodedText, tweet.entities.urls) : decodedText;
const text = tweet.entities.user_mentions && tweet.entities.user_mentions.length > 0 ? await replaceTwitterMentions(textReplaceURLs, tweet.entities.user_mentions) : textReplaceURLs;
const files: MiDriveFile[] = [];