"synchronous transformer must export a "process" function" when using processAsync

[bitttttten]

To Reproduce

module.exports = {
	async processAsync(src, filepath, config) {
		// ..
	},
}

When running on CI I get this error:

FAIL ./test.js
  ● Test suite failed to run

    Jest: synchronous transformer /home/runner/work/jest-transformer-mdx/jest-transformer-mdx/index.js must export a "process" function.

      at invariant (node_modules/@jest/transform/build/ScriptTransformer.js:1091:11)

You can see the CI result here, and you can view the transformer here.

Expected behavior

It matches what happens when I run the tests locally:

> [email protected] test:index
> jest --config jest.config.js

 PASS  ./test.js
  jest transformer md
    ✓ that frontMatter is correct (1 ms)
    ✓ that it still renders in react (21 ms)

Test Suites: 1 passed, 1 total
Tests:       2 passed, 2 total
Snapshots:   1 passed, 1 total
Time:        0.863 s, estimated 1 s
Ran all test suites.

Link to repl or repo (highly encouraged)

All the files are in the public repo: https://github.com/bitttttten/jest-transformer-mdx

envinfo

CI

  System:
    OS: Linux 5.4 Ubuntu 20.04.2 LTS (Focal Fossa)
    CPU: (2) x64 Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
  Binaries:
    Node: 14.17.0 - /opt/hostedtoolcache/node/14.17.0/x64/bin/node
    Yarn: 1.22.10 - /usr/local/bin/yarn
    npm: 6.14.13 - /opt/hostedtoolcache/node/14.17.0/x64/bin/npm
  npmPackages:
    jest: ^27.0.1 => 27.0.1 

Local machine

  System:
    OS: macOS 11.2.1
  Binaries:
    Node: 14.15.4 - ~/.nvm/versions/node/v14.15.4/bin/node
    Yarn: 1.22.10 - /usr/local/bin/yarn
    npm: 6.14.10 - ~/.nvm/versions/node/v14.15.4/bin/npm
  npmPackages:
    jest: ^27.0.1 => 27.0.1 

[bitttttten]

I see in #11226 that async transformations are only supported in ESM mode. So is there any way to use an async transformer without relying on users of the transformer to run jest in ESM mode?

My use case is that I am using mdx, which is async by default. So it's not really possible to write this transformer synchronously 🤔

[bitttttten]

I am still having no luck, even when trying to emulate the setup in your tests following this file.

// jest.config.js
	transform: {
		"^.+\\.js$": "babel-jest",
		"^.+\\.(md|mdx)$": [
			path.resolve(__dirname, "mdx-transformer.cjs"),
			{
				mdxOptions: {
					rehypePlugins: [rehypeSlug],
				},
			},
		],
	},

// mdx-transformer.cjs
"use strict"

const mdx = require("@mdx-js/mdx")
const { process: babelProcess } = require("babel-jest")

module.exports = {
	async processAsync(src, filepath, config) {
		const jsx = await mdx(withFrontMatter, { ...config?.transformerConfig?.mdxOptions, filepath })
		const toTransform = `import {mdx} from '@mdx-js/react';${jsx}`
		return babelProcess(toTransform, filepath, config).code
	},
}

I then run it with NODE_OPTIONS=--experimental-vm-modules npx jest --no-cache, and I still hit the Jest: synchronous transformer jest-transformer-mdx/mdx-transformer.cjs must export a "process" function.

I also played around with different ways of running Jest as per the ECMAScript Modules docs like node --experimental-vm-modules node_modules/.bin/jest etc. and I can see it is enabled:

(node:30693) ExperimentalWarning: VM Modules is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
 FAIL  ./test.js
  ● Test suite failed to run

    Jest: synchronous transformer jest-transformer-mdx/mdx-transformer.cjs must export a "process" function.

I am really trying to copy what is in your tests, am I missing something? Or is this a bug in jest?

Also I understand if this goes out of scope of this issue :)

[ahnpnl]

process is a fallback for processAsync so that’s why process is always required IIRC.

[bitttttten]

Do you mean to stub process? I had tried that with:

module.exports = {
	process() {
		return ""
	},
	async processAsync(src, filepath, config) {
		// omitted
	},
}

Which sadly doesn't work. It's using the return value from process 🤔

[ahnpnl]

Not sure if .cjs is the issue, what if you use .js for your transformer?

[SimenB]

I see in #11226 that async transformations are only supported in ESM mode. So is there any way to use an async transformer without relying on users of the transformer to run jest in ESM mode?

My use case is that I am using mdx, which is async by default. So it's not really possible to write this transformer synchronously 🤔

require is synchronous, so async transform is only supported when using ESM.

As for your following error, I assume it's because by default jest loads most files as CJS, you can use extensionsToTreatAsEsm as mentioned in the docs to tell jest that markdown files will be ESM.

You shouldn't need to define process if you transformer is only used asynchronously (i.e. in ESM mode)

[bitttttten]

I did come across extensionsToTreatAsEsm in the docs, however it noted that ".mjs" files were picked up already, so I tried renaming my transformer to mdx-transformer.mjs and refactoring it to support ESM.

import path from "path"
import matter from "gray-matter"
import stringifyObject from "stringify-object"
import mdx from "@mdx-js/mdx"
import babelJest from "babel-jest"

async function processAsync(src, filepath, config) {
	// omitted
}

export default { processAsync }

However that still gave me Jest: synchronous transformer jest-transformer-mdx/mdx-transformer.mjs must export a "process" function.. I tried including .md in the array also and got:

 FAIL  ./test.mdx-options.js
  ● Test suite failed to run

    Must use import to load ES Module: jest-transformer-mdx/test.md

      1 | import React from "react"
    > 2 | import Component, { frontMatter } from "./test.md"
        | ^

I think I am getting very stuck, I have tried:

• Transformer written in CommonJS, always gets must export a "process" function with or without VM Modules feature enabled, and without extensionsToTreatAsEsm: [".md"],

• Transformer written in CommonJS with extensionsToTreatAsEsm: [".md"], gets Must use import to load ES Module: jest-transformer-mdx/test.md

• Transformer written with ESM with extensionsToTreatAsEsm: [".md"] gets Must use import to load ES Module.

• Transformer written with ESM without extensionsToTreatAsEsm: [".md"] gets must export a "process" function.

Even though I know these combinations don't work, I just wanted to explore to see if I got any different errors. On paper, it should work with only exporting processAsync, in mdx-transformer.cjs, with VM Modules enabled as that is like you have it in your test files. I tried to follow the examples in your test files, although I hit similar issues too.

[SimenB]

The transformer can be either ESM or CJS, extensionsToTreatAsEsm is for the files that are transformed - i.e. you want extensionsToTreatAsEsm: ['md', 'mdx'] as you have.

You get Must use import to load ES Module since you use require to load it - use import or import() (from your code example, you probably transpile the JS to CJS)

[bitttttten]

What do you mean with "since you use require to load it"? Do you mean inside my test file? As in my test I import the file with import, like Import * as Test from "./test.md". Maybe I am missing something here, so sorry if I don't understand that :)

So I tried taking this step by step, and wrote the transformer with ESM. I could only get it running without getting Cannot use import statement outside a module errors unless the transformer file extension is .mjs. I can't seem to get this error to go away without renaming the file.

No problem though, now the transformer is written with ESM, with the file extension mjs. However now I still get the error from my original issue. If I use ESM with and run jest with NODE_OPTIONS=--experimental-vm-modules npx jest --no-cache --config jest.config.mdx-options-inline.js, then I get the Must use import to load ES Module error. I am kind of put off by this since it would mean users of the transformer would need to do that too.

With CJS, it runs fine but I also hit the same error from my original issue, synchronous transformer mdx-options.js must export a "process" function. Which I totally get, CJS does not support processAsync. I just wanted to make sure it was not my code or setup causing any odd issues!

[vadirn]

@bitttttten You might want to try adding "type": "module" to your package.json - this is what worked for me 😅

[Jessidhia]

I ran into this when trying to port a cascading transform from Jest 26 to Jest 27 because the ScriptTransformer export was removed from @jest/transform, and the package.json exports prevents the easy workaround. The old codepath would use a new ScriptTransformer to create a synchronous transformer, but as only the async createScriptTransformer is available now, it's no longer possible to chain transformers, at least not in the way this transformer (https://github.com/storybookjs/storybook/blob/next/addons/storyshots/storyshots-core/injectFileName.js) did it, so it seems I cannot (obviously) port it to Jest 27.

"it will be easy, I just need to change , config, { instrument } to , { config, instrument }" 😆

[sebastianrothe]

I see in #11226 that async transformations are only supported in ESM mode. So is there any way to use an async transformer without relying on users of the transformer to run jest in ESM mode?
My use case is that I am using mdx, which is async by default. So it's not really possible to write this transformer synchronously 🤔

require is synchronous, so async transform is only supported when using ESM.

As for your following error, I assume it's because by default jest loads most files as CJS, you can use extensionsToTreatAsEsm as mentioned in the docs to tell jest that markdown files will be ESM.

You shouldn't need to define process if you transformer is only used asynchronously (i.e. in ESM mode)

But this is just the module resolution, not the execution of the function. If this is true, then no one using a CJS library would be able to use an async function. Which I don't think is true.

[SimenB]

@Jessidhia We discussed some sort of async "setup" function, but ended up with "ESM is enough" (#11081 (comment)). Not sure if that would help in your case... Supporting some way of composing multiple transforms would be pretty nice. Could you open up a new issue with that use case?

---

@sebastianrothe I don't really follow what you're saying. Jest implementing JIT transforms, so require needs to remain synchronous - this has nothing to do with resolution at all. And you can import CJS code via import, so you should never be blocked by your libraries being CJS

[sebastianrothe]

@SimenB I was wondering, if an exclusive (processAsync only) asyncTransformer in CJS is possible. We maintain https://github.com/mihar-22/svelte-jester and replaced processwith processAsync in our transformer. We also switched the code to ESM, but now the users of the library have a lot of problems with the experimental flag required for ESM. So we thought about transpiling the ESM version to CJS for those users.

[SimenB]

That's not possible unfortunately as CJS cannot be asynchronous

[sebastianrothe]

That's not possible unfortunately as CJS cannot be asynchronous

But then this transformer https://github.com/facebook/jest/blob/master/e2e/transform/async-transformer/my-transform.cjs gets transpiled to ESM?

Ah, the test itself is ESM https://github.com/facebook/jest/blob/main/e2e/transform/async-transformer/__tests__/test.js

[CourtHive]

I have been waiting months to see whether this issue would be addressed. When I update to Babel/Jest 27 all of my almost 300 test suites fail with:

Jest: a transform must export a process function.

at ScriptTransformer._getTransformer (node_modules/jest-runner/node_modules/@jest/transform/build/ScriptTransformer.js:368:15)

https://github.com/CourtHive/tods-competition-factory

[SimenB]

I'll close this as the OP answered themselves in the first comment.

So is there any way to use an async transformer without relying on users of the transformer to run jest in ESM mode?

No, when a user calls require, the result must be returned synchronously, there is no way for that to ever by async. People will need to use ESM.

I'll add that you can use import() from within CJS, which is async, but the file that is imported will still need to be native ESM. But the use case if the OP (MDX) that can probably work - just transform to ESM rather than CJS (please see docs, you still need the runtime flag to node and extensionsToTreatAsEsm option).

---

@CourtHive feel free to open up a new issue with minimal reproduction steps.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.