Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Where can I find some sample about how to use bulk api to upload documents? #259

Closed
zhaoyi0113 opened this issue Jul 1, 2022 · 6 comments · Fixed by #277
Closed

Comments

@zhaoyi0113
Copy link

I am trying to use bulk API to upload many documents in one request. But I keep get Reason: request body is required" error response. Is there any document I can see how to set the parameter correctly?

I am using below code to upload the documents.

esClient.bulk({ body: bulkRecords });
@tymondragon
Copy link

@dblock
Copy link
Member

dblock commented Jul 20, 2022

I think @zhaoyi0113 is looking for a working sample that uses opensearch-js to call the bulk API. Anyone?

@ekeyser
Copy link

ekeyser commented Jul 29, 2022

This is what I did. It's javascript and it's quick and garbage code but maybe it'll help :-) Something about POST bulk indexes need to be separated by newlines? I'm not sure the full story behind this.

                  let strBulkBody = '';


                    batchItems.forEach((item) => {

                        strBulkBody += `{ "index": {"_index": "${index_name}"} }\n`;
                        strBulkBody += `${JSON.stringify(item)}\n`;
                    });


                    this.client.bulk({
                        body: strBulkBody,
                    })
                        .then((d) => {

                            resolve(d);
                        })
                        .catch((e) => {

                            reject(e);
                        });

I should add that I'm not 100% sure this works. It says it did but I just haven't queried the index yet. I also think this should be a core part of the documentation. Maybe I'll open a PR.

@harshavamsi
Copy link
Collaborator

@zhaoyi0113

Here's a code snippet to bulk upload documents.

    var bulk_documents = [
        {
            index: {
                _index: 'books-king',
                _id: '2'
            }
        },
        {
            title: 'IT',
            author: 'Stephen Kings',
            year: '1986',
        },
        {
            create: {
                _index: 'test',
                _id: '3'
            }
        },
        {
            title: 'The Green Mile',
            author: 'Stephen Kings',
            year: '1996',
        },
        {
            create: {
                _index: 'test',
                _id: '4'
            }
        },
        {
            title: 'Carrie',
            author: 'Stephen Kings',
            year: '1974',
        }
    ];

    var response = await client.bulk({ body: bulk_documents });
    console.log(response.body);

Will try to add some documentation in the GETTING_STARTED guide.

@harshavamsi
Copy link
Collaborator

Tracked in #277

@harshavamsi
Copy link
Collaborator

@wbeckler @VachaShah I believe safe to close this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants