You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am running into an issue with 'yield' for gerapy-playwright when I need to access a login page. I try to run:
yield PlaywrightRequest(login_page_url, self.parse_login, actions = self.login_action)
in order to first use playwright to login and then access data that can only be accessed when logging in with self.parse_login. I am getting a:
builtins.TypeError: is not JSON serializable.
It seems that the action is saved in the meta data as a method and cannot be passed to the scheduler. Is it possible to type cast the action as a string and then when the action is called later, to do a method call on the string? If I understand correctly, the action is produced on line 339 of the downloadermiddlewares.py inside of gerepy-playwright. Would it be possible to evaluate the string as a method so that the scrapy-cluster scheduler can pass the string but gerapy-playwright still calls the self.login_action method prior to the self.parse_login?
The text was updated successfully, but these errors were encountered:
Hello,
I am running into an issue with 'yield' for gerapy-playwright when I need to access a login page. I try to run:
yield PlaywrightRequest(login_page_url, self.parse_login, actions = self.login_action)
in order to first use playwright to login and then access data that can only be accessed when logging in with self.parse_login. I am getting a:
builtins.TypeError: is not JSON serializable.
I am using scrapy cluster along with gerapy-playwright in order to run a scheduler for all the spiders that I have: https://github.com/istresearch/scrapy-cluster
It seems that the action is saved in the meta data as a method and cannot be passed to the scheduler. Is it possible to type cast the action as a string and then when the action is called later, to do a method call on the string? If I understand correctly, the action is produced on line 339 of the downloadermiddlewares.py inside of gerepy-playwright. Would it be possible to evaluate the string as a method so that the scrapy-cluster scheduler can pass the string but gerapy-playwright still calls the self.login_action method prior to the self.parse_login?
The text was updated successfully, but these errors were encountered: