Slow speed

[IshanDindorkar]

Hello @argenisleon / @FavioVazquez -

I am running couple of operations like mentioned below -
df.rows.select(fbdt("id", "integer")).table()
df.rows.select(fbdt("id", "float")).table()

I am using notebook for performing these operations. As per console, the operation is in progress but it is slow in spitting out results. The dataset on which I am running operations has close to 90 million records. My question is there any configuration which I can use to speed up computation of spark job.
FYI, I am using an Ubuntu machine with 4 chores and 32 GB RAM.
Could you please advise.

Thank you for your support.

Best-
Ishan

[issue-label-bot]

Issue-Label Bot is automatically applying the label question to this issue, with a confidence of 0.69. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[argenisleon]

Hi,

Definitely this a slow operation. How much time is taking? Maybe we could explorer how to speed this up?

[IshanDindorkar]

It took 2-3 hours approximately for an individual operation.

[argenisleon]

Looking at the function https://github.com/ironmussa/Optimus/blob/master/optimus/functions.py#L398 we use fastnumbers library to speed things up when handling ints and floats. The function is super simple I am not sure if we have room for improvements here.

@FavioVazquez can you take a look on this?

[IshanDindorkar]

@argenisleon @FavioVazquez - did you guys made some changes to address this issue in the new release? Just curious!

[argenisleon]

Not at the moment. We are thinking about other approaches. Any thought about this?

[argenisleon]

We could improve performance If we implement #322

[argenisleon]

Hi @IshanDindorkar,

We rebuild the profiler to improve parallelism. In local mode, we cut the time by 1/3.
Hope this can help