Restart issue with background fields continues to exist

[PrometheusPi]

This is a follow up to #2113:

Test with the TWTS/plane wave background field setup (based on dev) still showed disappearing particles and wrong field values when used with restarts. Thus a restart issue still exists.

---

UPDATE/important to mention (thanks @ax3l for pointing this out)
This is (most likely) an issue with the field restart! The disappearing particles is the most obvious result, but is probably only caused by the wrong field values after restart.

UPDATE No. 2:
I forgot to cherry-pick #2112 aee7784672da6c3ded6fe00f47be9fdcd4b97948 - I know updated the plots and the error is about a factor 2 in the simple setup presented below.

---

The following difference could be seen for the same simulation setup - fist with restart every 10kth iteration, the second without any restart.

As mentioned by @psychocoderHPC in #2113 I will try to find an easy setup to reproduce this bug based on 0.3.0.

mentioning:
@BeyondEspresso
@steindev

[ax3l]

@PrometheusPi checkpoint/restart with ADIOS or HDF5?

[PrometheusPi]

@ax3l Restart with HDF5.

[PrometheusPi]

@psychocoderHPC I could reproduce this error using 0.3.0 with a cherry picked daf0d002704b66abf6192995e95f5f7cfc73a30e (#2113):

---

UPDATE No. 2:

• update pictures - error now much larger
• cherry-picked the plugin now, too fix plugin energy field reduce #2112

---

Based on the default Laser Wakefield example (8 GPUs), but without particles and without Laser I used the following background field:

namespace picongpu
{
    class FieldBackgroundE
    {
    public:
        /* Add this additional field for pushing particles */
        static constexpr bool InfluenceParticlePusher = true;

        /* We use this to calculate your SI input back to our unit system */
        PMACC_ALIGN(m_unitField, const float3_64);

        HDINLINE FieldBackgroundE( const float3_64 unitField ) : m_unitField(unitField)
        {}

        /** Specify your background field E(r,t) here
         *
         * \param cellIdx The total cell id counted from the start at t = 0
         * \param currentStep The current time step */
        HDINLINE float3_X
        operator()( const DataSpace<simDim>& cellIdx,
                    const uint32_t currentStep ) const
        {
            /* example: periodicity of 20 microns ( = 2.0e-5 m) */
            constexpr float_64 period_SI(20.0e-6);
            constexpr float_64 omega_SI = 2.0 * PI * 2.9979e8 / period_SI;
            /* calculate cells -> SI -> m to microns*/
            const float_64 y_SI = cellIdx.y() * SI::CELL_HEIGHT_SI;
            /* note: you can also transform the time step to seconds by
             *       multiplying with DELTA_T_SI */

            /* specify your E-Field in V/m and convert to PIConGPU units */
            const float_X sinArg = precisionCast<float_X>( y_SI / period_SI * 2.0 * PI - currentStep *SI::DELTA_T_SI * omega_SI);
            return float3_X(0.0, math::sin( sinArg ) / m_unitField[1], 0.0);
        }
    };

and equivalent for the magnetic field.

There is an obvious (but in this case small) difference in the field energy:

Due to the periodic nature of these spikes, I will re-run a simulation without moving window and see if the spikes disappear.

[PrometheusPi]

This bug persist even without moving window:

---

Update No. 2:
Update after cherry-pick -error now much larger

---

[PrometheusPi]

The error does not occur for normal/standard laser input (in this case plane wave - with periodic (transversal) boundary conditions and moving window) which is treated by the Yee solver.

[ax3l]

I could reproduce this error using 0.3.0 with a cherry picked daf0d00 (#2113):

careful, #2113 is not part of the last release and might actually cause the problem, when testing 0.3.0 please only cherry-pick #2112

[PrometheusPi]

Without the cherry-picked commit from the bug fix #2113 - the same error occurs with restart.

[ax3l]

but looks less severe, probably 0.3.0 just needs the step-1u in the Sub() functors to be fixed...

[psychocoderHPC]

I thing I know where the field coming from and will create a bugfix.

[psychocoderHPC]

@PrometheusPi can we close this issue

[PrometheusPi]

Yes, we can close this issue.

[psychocoderHPC]

solved with #2139