Skip to content

Commit

Permalink
Prevent newlines in JSONL
Browse files Browse the repository at this point in the history
If a set of 100 ids are hydrated and not one of them is still available
the Hydrator was writing a blank line to the jsonl file. This could
cause problems for downstream users of the jsonl that are expecting each
line to contain a JSON object.

Refs #48
  • Loading branch information
edsu committed Jun 28, 2020
1 parent 40bc882 commit 6cde028
Show file tree
Hide file tree
Showing 2 changed files with 5 additions and 3 deletions.
6 changes: 4 additions & 2 deletions app/utils/twitter.js
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,10 @@ export async function getUserSettings(auth) {
export async function hydrateToStream(ids, out, auth, event, datasetId, tries=0) {
try {
const tweets = await fetchTweets(ids, auth)
const text = tweets.map(t => JSON.stringify(t)).join('\n')
out.write(text + "\n")
if (tweets.length > 0) {
const text = tweets.map(t => JSON.stringify(t)).join('\n')
out.write(text + "\n")
}
event.sender.send(UPDATE_PROGRESS, {
datasetId: datasetId,
idsRead: ids.length,
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "hydrator",
"version": "0.0.11",
"version": "0.0.12",
"description": "A desktop utility to hydrate Twitter ID datasets.",
"main": "init.js",
"author": {
Expand Down

0 comments on commit 6cde028

Please sign in to comment.