Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to generate images in the Media Inserter #535

Merged
merged 8 commits into from
Aug 10, 2023
Merged

Add ability to generate images in the Media Inserter #535

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
28ff330
Initial work done on testing out the new registerInserterMediaCategor…
dkotter Jul 13, 2023
695a69e
Merge branch 'develop' into feature/inserter-media-category
dkotter Jul 13, 2023
d6d05ed
Remove code that is no longer needed. Refactor our JS a bit to only l…
dkotter Jul 13, 2023
fd6f437
Remove hardcoded API response
dkotter Jul 13, 2023
30d14bd
Update script name
dkotter Jul 13, 2023
62299ca
Fix lint issue
dkotter Jul 13, 2023
d220238
Modify our inserter code a bit to use apiFetch
dkotter Jul 28, 2023
ff29908
Refactor how we register the media category by copying what Jetpack d…
dkotter Jul 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions includes/Classifai/Providers/OpenAI/DallE.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,14 @@ public function enqueue_admin_scripts( $hook_suffix = '' ) {
true
);

wp_enqueue_script(
'classifai-inserter-media-category',
CLASSIFAI_PLUGIN_URL . 'dist/inserter-media-category.js',
get_asset_info( 'inserter-media-category', 'dependencies' ),
get_asset_info( 'inserter-media-category', 'version' ),
true
);

/**
* Filter the default attribution added to generated images.
*
Expand Down
67 changes: 67 additions & 0 deletions src/js/gutenberg-plugins/inserter-media-category.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
/**
* Some code here was copied from Jetpack's implementation of the inserter media category.
* See https://github.com/Automattic/jetpack/pull/31914
*/
import apiFetch from '@wordpress/api-fetch';
import { dispatch, select, subscribe } from '@wordpress/data';
import { __ } from '@wordpress/i18n';
import { addQueryArgs } from '@wordpress/url';

const { classifaiDalleData } = window;

const isInserterOpened = () =>
select( 'core/edit-post' )?.isInserterOpened() ||
select( 'core/edit-site' )?.isInserterOpened() ||
select( 'core/edit-widgets' )?.isInserterOpened?.();

const waitFor = async ( selector ) =>
new Promise( ( resolve ) => {
const unsubscribe = subscribe( () => {
if ( selector() ) {
unsubscribe();
resolve();
}
} );
} );

waitFor( isInserterOpened ).then( () =>
dispatch( 'core/block-editor' )?.registerInserterMediaCategory?.(
registerGenerateImageMediaCategory()
)
);

const registerGenerateImageMediaCategory = () => ( {
name: 'classifai-generate-image',
labels: {
name: classifaiDalleData.tabText,
search_items: __( 'Enter a prompt', 'classifai' ),
},
mediaType: 'image',
fetch: async ( { search = '' } ) => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this needs to be debounced a little, it looks like it makes multiple requests while typing in the prompt.

Screen Shot 2023-08-02 at 9 34 16 am

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mentioned that in the PR description that if someone doesn't type super fast, it will make multiple requests. I'm struggling with figuring out how to add debouncing here, since debouncing already happens on the WordPress side (see https://github.com/WordPress/gutenberg/blob/trunk/packages/block-editor/src/components/inserter/hooks/use-debounced-input.js). Everything I've tried to add another layer of debouncing isn't working. Either I'm not sure how to make this work (very likely) or this may not be possible since the only thing we have control over is the actual fetch function, not when that function gets called.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As an example, if I add debounce around our fetch function:

fetch: debounce( async ( { search = '' } ) => {
    ...
}, 1000 ),

and leave everything else the same, it does properly debounce the requests but it does not properly render the results. It seems to always be one request behind. So if I enter an image prompt and wait for a second or two (for the debounce to kick in), I do see my request being made but no results are rendered. If I then change the image prompt, the previous results render immediately and then once the wait period is up, a new request is made (but the results from that aren't rendered unless I make changes to the prompt again).

Again, this may very well be something I'm doing wrong but could also be some limitations around how Core is doing the rendering here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry Darin, I missed this in the description.

As the feature didn't make it in to WP 6.3, is it worth adding a note that it requires the Gutenberg plugin/WP 6.4 or later to the changelog entry?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm surprised this didn't make it into WP 6.3 as this changed was added to Gutenberg in min-June. I think this is probably worth holding off on until this comes to WordPress itself, at least that's my opinion.

I also think it's worth discussing a bit more around how requests are made here. Since image generation requests are expensive, it's not great that you'll almost certainly end up making multiple requests as you enter an image prompt. I'm wondering if it's worth opening an issue on the Gutenberg side to see if we can get this feature modified a bit? I would love to be able to toggle between search happening as you type (the way it works now) or the ability to only search if someone clicks a submit button (which would solve our needs). Or if we can filter the default debounce value to make that higher, that also would help.

Not sure if either of those are changes they'll deem worth adding but may be a good conversation to start.

if ( ! search ) {
return [];
}

const images = await apiFetch( {
path: addQueryArgs( classifaiDalleData.endpoint, {
prompt: search,
format: 'b64_json',
} ),
method: 'GET',
} )
.then( ( response ) =>
response.map( ( item ) => ( {
title: search,
url: `data:image/png;base64,${ item.url }`,
previewUrl: `data:image/png;base64,${ item.url }`,
id: undefined,
alt: search,
caption: classifaiDalleData.caption,
} ) )
)
.catch( () => [] );

return images;
},
isExternalResource: true,
} );
3 changes: 3 additions & 0 deletions webpack.config.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@ module.exports = {
],
'post-excerpt': [ './src/js/post-excerpt/index.js' ],
'media-modal': [ './src/js/media-modal/index.js' ],
'inserter-media-category': [
'./src/js/gutenberg-plugins/inserter-media-category.js',
],
'generate-title-classic': [
'./src/js/openai/classic-editor-title-generator.js',
],
Expand Down