Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add improvements to translation pipeline #3555

Merged
merged 11 commits into from
May 27, 2021
15 changes: 15 additions & 0 deletions bin/i18n-download.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
function pFail() {
if [ -n "$1" ]
then
echo "Message: $1"
fi
echo "[KO]"
exit 1
}

if [ -n "${REFRESH_I18N}" ] ; then
node src/i18n-cache/index.js $1 || pFail
fi
# Overwrite the i18n cache of react-native-editor package
cp -r src/i18n-cache/gutenberg/data gutenberg/packages/react-native-editor/i18n-cache || pFail
cp -r src/i18n-cache/gutenberg/index.native.js gutenberg/packages/react-native-editor/i18n-cache || pFail
5,818 changes: 2,908 additions & 2,910 deletions bundle/ios/App.js

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion bundle/ios/App.js.map

Large diffs are not rendered by default.

8 changes: 5 additions & 3 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,9 @@
"prern-bundle": "patch-package --patch-dir gutenberg/packages/react-native-editor/metro-patch",
"rn-bundle": "react-native bundle",
"postrn-bundle": "patch-package --reverse --patch-dir gutenberg/packages/react-native-editor/metro-patch",
"bundle": "npm run clean; npm run core prebundle && npm run bundle:js && npm run genstrings",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to note that I removed npm run core prebundle from this command as now we're updating the i18n cache via the new i18n:gutenberg command.

"i18n:gutenberg": "./bin/i18n-download.sh gutenberg",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the future, once the missing strings issue is solved, we would change this command to cross-env REFRESH_I18N=1 npm run i18n:gutenberg. This way we would enforce the cache update when generating the bundle.

"bundle": "npm run clean; npm run bundle:js && npm run genstrings",
"prebundle:js": "npm run i18n:gutenberg",
"bundle:js": "npm run bundle:android && npm run bundle:ios",
"bundle:android": "npm run bundle:android:text && npm run bundle:android:bytecode",
"bundle:android:text": "mkdir -p bundle/android && npm run rn-bundle -- --platform android --dev false --entry-file ./index.js --assets-dest ./bundle/android --bundle-output ./bundle/android/App.text.js --sourcemap-output ./bundle/android/App.text.js.map",
Expand All @@ -73,9 +75,9 @@
"makepot:android": "npm run wp -- i18n make-pot ./gutenberg/packages --include=*.native.js,*.android.js --exclude=test/*,e2e-tests/*,build/*,build-module/*,build-style/*,*.js,*.php --subtract=./gutenberg.pot --ignore-domain gutenberg-android.pot",
"makepot:ios": "npm run wp -- i18n make-pot ./gutenberg/packages --include=*.native.js,*.ios.js --exclude=test/*,e2e-tests/*,build/*,build-module/*,build-style/*,*.js,*.php --subtract=./gutenberg.pot --ignore-domain gutenberg-ios.pot",
"premakepot:gutenberg": "npm run clean:gutenberg && npm run build:gutenberg",
"makepot:gutenberg": "npm run wp -- i18n make-pot ./gutenberg/build --ignore-domain gutenberg.pot",
"makepot:gutenberg": "npm run wp -- i18n make-pot ./gutenberg/packages --exclude=test/*,e2e-tests/*,build/*,build-module/*,build-style/*,*.native.js,*.ios.js,*.android.js,bundle/* --ignore-domain gutenberg.pot",
"postmakepot:gutenberg": "npm run clean:gutenberg",
"pregenstrings": "test -f gutenberg.pot || npm run makepot:gutenberg",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The gutenberg.pot file can be different if new strings are introduced so I think it's not reliable to prevent generating it if it already exists.

"pregenstrings": "npm run makepot:gutenberg",
"genstrings": "npm run genstrings:android && npm run genstrings:ios",
"genstrings:android": "npm run makepot:android && ./bin/po2android.js gutenberg-android.pot bundle/android/strings.xml",
"genstrings:ios": "npm run makepot:ios && ./bin/po2swift.js gutenberg-ios.pot bundle/ios/GutenbergNativeTranslations.swift",
Expand Down
1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/ar.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/bg.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/bo.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/ca.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/cs.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/cy.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/da.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/de.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/el.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/en-au.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/en-ca.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/en-gb.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/en-nz.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/en-za.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/es-ar.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/es-cl.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/es-cr.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/es.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/fa.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/fr.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/gl.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/he.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/hr.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/hu.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/id.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/is.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/it.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/ja.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/ka.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/ko.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/nb.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/nl-be.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/nl.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/pl.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/pt-br.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/pt.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/ro.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/ru.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/sk.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/sq.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/sr.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/sv.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/th.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/tr.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/uk.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/ur.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/vi.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/zh-cn.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/i18n-cache/gutenberg/data/zh-tw.json

Large diffs are not rendered by default.

58 changes: 58 additions & 0 deletions src/i18n-cache/gutenberg/index.native.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
/* THIS IS A GENERATED FILE. DO NOT EDIT DIRECTLY. */
/* eslint-disable prettier/prettier */

const translations = {
"ar": require( "./data/ar.json" ),
"bg": require( "./data/bg.json" ),
"bo": require( "./data/bo.json" ),
"ca": require( "./data/ca.json" ),
"cs": require( "./data/cs.json" ),
"cy": require( "./data/cy.json" ),
"da": require( "./data/da.json" ),
"de": require( "./data/de.json" ),
"en-au": require( "./data/en-au.json" ),
"en-ca": require( "./data/en-ca.json" ),
"en-gb": require( "./data/en-gb.json" ),
"en-nz": require( "./data/en-nz.json" ),
"en-za": require( "./data/en-za.json" ),
"el": require( "./data/el.json" ),
"es": require( "./data/es.json" ),
"es-ar": require( "./data/es-ar.json" ),
"es-cl": require( "./data/es-cl.json" ),
"es-cr": require( "./data/es-cr.json" ),
"fa": require( "./data/fa.json" ),
"fr": require( "./data/fr.json" ),
"gl": require( "./data/gl.json" ),
"he": require( "./data/he.json" ),
"hr": require( "./data/hr.json" ),
"hu": require( "./data/hu.json" ),
"id": require( "./data/id.json" ),
"is": require( "./data/is.json" ),
"it": require( "./data/it.json" ),
"ja": require( "./data/ja.json" ),
"ka": require( "./data/ka.json" ),
"ko": require( "./data/ko.json" ),
"nb": require( "./data/nb.json" ),
"nl": require( "./data/nl.json" ),
"nl-be": require( "./data/nl-be.json" ),
"pl": require( "./data/pl.json" ),
"pt": require( "./data/pt.json" ),
"pt-br": require( "./data/pt-br.json" ),
"ro": require( "./data/ro.json" ),
"ru": require( "./data/ru.json" ),
"sk": require( "./data/sk.json" ),
"sq": require( "./data/sq.json" ),
"sr": require( "./data/sr.json" ),
"sv": require( "./data/sv.json" ),
"th": require( "./data/th.json" ),
"tr": require( "./data/tr.json" ),
"uk": require( "./data/uk.json" ),
"ur": require( "./data/ur.json" ),
"vi": require( "./data/vi.json" ),
"zh-cn": require( "./data/zh-cn.json" ),
"zh-tw": require( "./data/zh-tw.json" ),
};

export const getTranslation = ( locale ) => translations[ locale ];

/* eslint-enable prettier/prettier */
172 changes: 172 additions & 0 deletions src/i18n-cache/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,172 @@
/* eslint-disable no-console */

/**
* External dependencies
*/
const fs = require( 'fs' );
const path = require( 'path' );
const fetch = require( 'node-fetch' );

const supportedLocales = [
'ar', // Arabic
'bg', // Bulgarian
'bo', // Tibetan
'ca', // Catalan
'cs', // Czech
'cy', // Welsh
'da', // Danish
'de', // German
'en-au', // English (Australia)
'en-ca', // English (Canada)
'en-gb', // English (UK)
'en-nz', // English (New Zealand)
'en-za', // English (South Africa)
'el', // Greek
'es', // Spanish
'es-ar', // Spanish (Argentina)
'es-cl', // Spanish (Chile)
'es-cr', // Spanish (Costa Rica)
'fa', // Persian
'fr', // French
'gl', // Galician
'he', // Hebrew
'hr', // Croatian
'hu', // Hungarian
'id', // Indonesian
'is', // Icelandic
'it', // Italian
'ja', // Japanese
'ka', // Georgian
'ko', // Korean
'nb', // Norwegian (Bokmål)
'nl', // Dutch
'nl-be', // Dutch (Belgium)
'pl', // Polish
'pt', // Portuguese
'pt-br', // Portuguese (Brazil)
'ro', // Romainian
'ru', // Russian
'sk', // Slovak
'sq', // Albanian
'sr', // Serbian
'sv', // Swedish
'th', // Thai
'tr', // Turkish
'uk', // Ukrainian
'ur', // Urdu
'vi', // Vietnamese
'zh-cn', // Chinese (China)
'zh-tw', // Chinese (Taiwan)
];

const getLanguageUrl = ( locale, plugin ) =>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file is almost the same as the one we have in gutenberg/packages/react-native-editor/i18n-cache but here I included the plugin parameter that will be useful for when we introduce the translations for Jetpack.

`https://translate.wordpress.org/projects/wp-plugins/${ plugin }/dev/${ locale }/default/export-translations\?format\=json`;

const getTranslationFilePath = ( locale ) => `./data/${ locale }.json`;

const fetchTranslation = ( locale, plugin ) => {
console.log( 'fetching', getLanguageUrl( locale, plugin ) );
const localeUrl = getLanguageUrl( locale, plugin );
return fetch( localeUrl )
.then( ( response ) => response.json() )
.then( ( body ) => {
return { response: body, locale };
} )
.catch( () => {
console.error( `Could not find translation file ${ localeUrl }` );
} );
};

const fetchTranslations = ( plugin ) => {
const fetchPromises = supportedLocales.map( ( locale ) =>
fetchTranslation( locale, plugin )
);

// Create data folder if it doesn't exist
const dataDir = path.join( __dirname, `${ plugin }/data` );
if ( ! fs.existsSync( dataDir ) ) {
fs.mkdirSync( dataDir );
}

return Promise.all( fetchPromises ).then( ( results ) => {
const fetchedTranslations = results.filter( Boolean );
const translationFilePromises = fetchedTranslations.map(
( languageResult ) => {
return new Promise( ( resolve, reject ) => {
const translationRelativePath = getTranslationFilePath(
languageResult.locale,
plugin
);

const translationAbsolutePath = path.resolve(
__dirname,
`${ plugin }/${ translationRelativePath }`
);

fs.writeFile(
translationAbsolutePath,
JSON.stringify( languageResult.response ),
'utf8',
( err ) => {
if ( err ) {
reject( err );
} else {
languageResult.path = translationRelativePath;
resolve( translationRelativePath );
}
}
);
} );
}
);
return Promise.all( translationFilePromises ).then(
() => fetchedTranslations
);
} );
};

// if run as a script
if ( require.main === module ) {
const args = process.argv.slice( 2 );
const plugin = args[ 0 ] || 'gutenberg';
const pluginDir = path.join( __dirname, plugin );

if ( ! fs.existsSync( pluginDir ) ) {
fs.mkdirSync( pluginDir );
}

fetchTranslations( plugin ).then( ( translations ) => {
const indexNative = `/* THIS IS A GENERATED FILE. DO NOT EDIT DIRECTLY. */
/* eslint-disable prettier/prettier */

const translations = {
${ translations
.filter( Boolean )
.map(
( translation ) =>
`\t"${ translation.locale }": require( "${ translation.path }" ),`
)
.join( '\n' ) }
};

export const getTranslation = ( locale ) => translations[ locale ];

/* eslint-enable prettier/prettier */
`;

fs.writeFile(
path.join( pluginDir, 'index.native.js' ),
indexNative,
'utf8',
( error ) => {
if ( error ) {
console.error( error );
return;
}
console.log( 'Done.' );
}
);
} );
}

/* eslint-enable no-console */