Feature request: Import from CSV by generating json data

[evrim]

Hi there,

I must admit, I am happy with the status of the listmonk project. However, a missing feature is pretty much blocking the integration.

So, the import section requires/looks for json data for custom attributes. However, most of my clients only have access to programs such as spreadsheets (excel, ooffice, gdocs etc.) and it is pretty much not viable to tell them about json and friends.

So my questions is, is it possible to modify the 'import subscribers' functionality as defined in the following:

Data: A csv or some other tabular data file,
Function #1: Look for name, email columns, import them as usual,
Function #2: Import all other columns by converting them to json and assuming them as attributes.

This is crucial as I haven't found any method to make users life easier. This will make online users of collabora/onlyoffice to integrate with listmonk.

best,
evrim.

[benstaf]

I agree that a feature that would be nice to have for basic use is the 'attribute' column to accept a string, and not only a JSON, which is overkill in many cases

Because your suggestion seems overly complicated: if then the users export to CSV the data that they imported, they will get a column 'attributes' with JSON data.

(Unless the export function is modified to blow up the JSON in separate columns. This will still be unpractical in the general case, because JSON data can be nested in different depth levels, so JSON will remain)

In my use-case, I perform a separate pre-processing step to transform strings into JSONs.

[evrim]

Hi there,

Thanks for the reply. My suggestion is not overly complicated, just a checkbox to "consider other columns as json data" data and that's all. Pretty much removes the restriction of the need of a preprocessing stage which mite be a secondary use-case for me.

Anyway, thnx.

[knadh]

Hi @evrim.

Adding support for reading flat columns as JSON attributes has multiple consequences:

• Nested attributes can't be represented.
• Field types are completely lost--everything becomes a string.
• What if "attributes" and other fields are present? Now the system has to warn users to either include "attributes" OR arbitrarily named fields.
• Currently, order of fields are not mandated and "attributes" is optional. For arbitrary series of fields to be supported, a specific order has to be mandated. This already breaks backwards compatibility.
• Now if order is mandated and a new mandatory field is added as a part of the spec tomorrow, further breaks this behaviour and can potentially create a conflict between the new field name and arbitrary field names that users may have started using in their CSVs.

The ideal solution here is for you to write a simple script or tool that accepts a CSV in the format that is suitable for your usecase, and convert that to listmonk's specific format before uploading.

[benstaf]

Hi there,

Thanks for the reply. My suggestion is not overly complicated, just a checkbox to "consider other columns as json data" data and that's all. Pretty much removes the restriction of the need of a preprocessing stage which mite be a secondary use-case for me.

Anyway, thnx.

If you really need this feature, what you can do is code a special-purpose web app to import CSV files the way you want, transform them into Listmonk format, and then send them to Listmonk through the import API : https://listmonk.app/docs/apis/import/#post-apiimportsubscribers

For example, this web app could be based on this video tutorial (in Laravel PHP), which also has code and blog:

https://youtu.be/STJV2hTO1Zs

[kawie]

I agree this is one of the major drawbacks of Listmonk and super inconvenient for end-users, so I'm sad to see that @knadh decided not to offer this (even though I understand the arguments made in your posts).

In the meantime, I made a python script you can use to do preprocessing: https://gist.github.com/kawie/40cde96369a402ae61e72f54523c61d7