GSoCs Assemble!

[xtalax]

Hi all,

Many of you have reached out to me or @ChrisRackauckas about doing a GSoC related to NeuralPDE.jl, or PDE discretizers and we're both very happy to have you interested in these tools.

Some of you have already started working on the package, and that's great to see. Start small, and move on to bigger and more ambitious things as the summer progresses. A lot of hands make light work, but we're going to have to co-ordinate to avoid stepping on each other's toes and duplicating work.

If you are already working on something right now, please comment here introducing yourself to everyone, with a link to the issue describing what you are working on. If there is no issue yet describing what you are working on, please create one with some detail about your intended project and link it.

If you are interested in doing a GSoC either here or across the SciML PDEs ecosystem, please reach out and introduce yourself here with a little bit about what you are interested in and we can find something cool and useful for you to work on.

Finally, to all you lurkers who might be on the fence: Yes, You! Writing open source code can be extremely rewarding, and speaking from experience can lead to great and fulfilling careers. I started contributing in the summer of 2019, and I can tell you it has been a blast so far. So please do reach out if you have any interest in PDEs, SciML, Julia, Symbolic compilers or just math in general, and we'll find something your speed for you to work on.

A bit about me, I work as the lead dev on MethodOfLines.jl, and have recently taken over here too. We're going to expand the system to be one of the most general PDE solvers out there, and with very natural mathematical syntax, so there is plenty to do.

I Look Forward to Hearing from You!

Please comment:

• Your Name
• Your Major
• What you are interested in
• Any issues you are currently working on if any
• What you want to work on if you have some ideas

[AstitvaAggarwal]

Good Day! @xtalax @ChrisRackauckas my name is Astitva Aggarwal. I am Majoring in Computer Engineering. Im Interested in PINNs broadly and like working on experimental projects, Ive been working on issue #640 and have created a PR #666 for a new Feature in NNODE ( have also done some minor contributions before). I'm currently looking into MCMC based parameter estimation whenever I find some time, after being suggested to look into MCMC based approaches to Optimization(could be implemented using Turing.jl) for a GSOC project, I'm still learning about this field so I'm even finding many aspects from training strategies to Neural Network architectures really interesting.

[xtalax]

In terms of network architectures, I have recently seen the SIREN NN which seems like it promises to solve a lot of the problems (low pass behaviour) of neural networks, would be good to fully test, benchmark and integrate these in to the package including docs
https://arxiv.org/abs/2006.09661
https://docs.juliahub.com/Sophon/MOjGB/0.2.3/

[creataur]

Hi, I'm Pankaj Sharma.
I have graduated BSc (H.), MSc, and Post-MSc in Physics.
I am interested in "Enhancement of existing tools for discretizing PDEs. The finite differencing (FDM) library MethodOfLines.jl could be enhanced to allow non-uniform grids or compositions of operators. The finite element method (FEM) library FEniCS.jl could wrap more of the FEniCS library."
I am a novice in Julia, so I'll be learning the language for GSoC since I am highly interested in the application aspect of the project.
What I bring to the table is my training in PDEs, a persistent attitude when tackling new problems, and good experience in programming using c++. During my MSc Project, I used c++ for modeling Electromagnetic Wave Propagation through Optical Waveguides. In the project, I used Crank-Nicolson Method for solving PDEs, RK4 methods for 2nd Order ODEs in 2-Dimensions, and Thomas Method for solving tridiagonal linear systems.
I am looking forward to working with you guys. Thanks!

[sphinx-tech]

Hi everyone! I'm Sagar, a senior undergrad majoring in Mathematics And Computation.
Lately, Inverse problem solving through PiNNs has captured my attention (mainly Helmholtz scatterers). I have been working on Inverse Dirichlet Adaptive Loss - #670 and Neural Tangent Kernel Adaptive Loss - #501. I have created a quick PR for implementing these adaptive loss techniques #673.
Apart from these I am figuring out PDE Solvers for general domains and Complex geometries.

In the context of GSOC 23, my interests lie in Implementing enhanced meshers and training strategies for NeuralPDE.jl which respect the domain. This discussion seems to direct towards a viable solution (by working with .stl files and compatible frameworks) but it requires more brevity and flexibility.
Would love to hear and discuss with anyone who finds this interesting !

[sdesai1287]

Hi all, my name is Samedh Desai and I am a third year undergraduate student majoring in Computer Science and Economics at the University of Washington, Seattle. I have worked so far on creating my own training strategy, WeightedSampleTraining, which generates points for training based on the given inputs of weights and samples. It does this by first splitting the timespan into equal segments based on the number of weights (the length of the weights array), then sampling points in each segment based on that segment's corresponding weight, such that the total number of sampled points is equivalent to the given samples. This is useful for challenging differential equations problems, where a “strange” configuration of sampling points would train the neural network to fit the problem better. One case that I tested this method on was the Lotka-Volterra problem, where I was able to get reliable, accurate results that far surpassed outcomes for other methods. Here is a link to my PR: 635. I am working on finalizing this by cleaning up some details. For the future, I am interested in working on NeuralPDE.jl to improve training strategies for ODEs and also potentially building a new PDE parsing tool using symbolics.jl and some other discretizer packages.

[YichengDWu]

My concern is the correctness of NeuralPDE.jl. In my view, NeuralPDE has been incorrectly implemented for a long time, such as the serious errors I pointed out in two PRs:
#608
#610
For adaptive algorithms, my impression is that most implementations are incorrect, for example, MiniMaxAdaptiveLoss should assign a weight to each data point, rather than on the PDE loss and boundary condition level.

The adaptive loss algorithm implementation in NeuralPDE.jl is very ugly, making it difficult to verify the correctness of the implementation. Or a high-level algorithm may not achieve the desired effect due to low-level errors.

[YichengDWu]

By the way, I'd like to point out one more mistake: the loss function should not resample data points with every call, as this severely affects the convergence of BFGS:
https://github.com/SciML/NeuralPDE.jl/blob/master/src/training_strategies.jl#L105

[ChrisRackauckas]

I did do a big audit of parts of the library summer and agree it doesn't live up to the normal standards that we have in SciML, which is one of the reasons it's a big focus of this summer. The adaptive algorithms is a piece I did not audit in full which is why it's one of the projects for this year. And @xtalax is leading a charge on rewriting the parser to clean that up. I think those are the main parts I have concerns with.

That's part of the reason for improving NNODE with this. It's much easier to see the code in full on NNODE. This would then give us a clean test case that the PDE discretizations are doing the right thing since in constrained scenarios and the same starting weights they should do exactly the same thing. I want to start building regression tests around that.

[ChrisRackauckas]

By the way, I'd like to point out one more mistake: the loss function should not resample data points with every call, as this severely affects the convergence of BFGS:

No it's not a mistake, but it should be an option. Not resampling leads to some inaccuracies in the trained result in many cases if the points are not close enough. This is something seen in GridSampling. But yes BFGS needs a non-stochastic loss function. So StochasticTraining needs a resampling argument for controlling this depending on the context (which QuasiRandomTraining already has https://github.com/SciML/NeuralPDE.jl/blob/master/src/training_strategies.jl#L143). And it would be good to setup a trait in Optimization.jl so that it automatically can throw a nice error.

[YichengDWu]

Sorry I need add some context here. I'm convinced that it is a mistake to do minibatch & Adam in PINNs as pointed out here:

https://youtu.be/qYmkUXH7TCY?t=2938

Of course, it can work on a few toy equations, but it almost never works on more complex PDEs. In my personal experience, I always use (L)BFGS directly and perform resampling after it has optimized to a certain stage. Since the number of iterations is relatively small, the entire training process might not necessarily be slower than Adam.

[ChrisRackauckas]

I don't disagree. The benchmarks show pretty clearly that it's the best way to do things (example https://docs.sciml.ai/SciMLBenchmarksOutput/stable/PINNOptimizers/hamilton_jacobi/). And with some improvements to differentiability of Integrals.jl, most lower dimensional PDE cases should probably be using the QuadratureTraining strategy with BFGS (since HCubature isn't stochastic).

However, NeuralPDE.jl is a library. The defaults and tutorials should lead people to use the right things, but the options should be there to do the bad stuff. One of the biggest use cases of that of course is benchmarking, where making it a feature to use ADAM is how we can make benchmarks showing that it's bad.

[YichengDWu]

I completely agree. What I meant was that data sampling should not be done inside the loss function. Instead, it should be exposed to the user side, allowing users to pass the dataset into the loss function. This way, users have more options, such as whether to resample or even to perform non-uniform sampling, which is important for resampling algorithms.

[ChrisRackauckas]

I think we can have a CustomSampling algorithm or something that requires the user pass the sampling functions. Most people should probably be using a simple default though, and we can keep improving them. But yes there are cases where domain knowledge can give an improved sampler and adaptive sampling (i.e. MISER and VEGAS via QuadratureStrategy) can take a bit to adapt.

[YichengDWu]

I've made revisions to many aspects of NeuralPDE, for example a cleaner interface for adaptive loss:

https://github.com/YichengDWu/Sophon.jl/blob/main/docs/src/tutorials/sod.jl#L39

Users only need to concentrate on implementing the function (phi, x, ps) -> point-wise weight. I hope this can serve as a source of inspiration.

[ChrisRackauckas]

Yes that looks good to upstream.

[YichengDWu]

Right now, I'm finding it tricky to rework the code without messing up the many adaptive weight algorithms that are built on our current interface. Honestly, I'm having a hard time wrapping my head around this part of the source code. Maybe one day we'll just have to bite the bullet and rewrite NeuralPDE.jl from the ground up.

A potentially helpful suggestion would be to compare the training results between Sophon.jl and NeuralPDE.jl might help us spot where problems are hiding, is it in the new algorithm we're trying to implement or is it something in NeuralPDE itself?

[xtalax]

@YichengDWu What is the state of completeness in Sophon, could it be a better basis to rebuild from?

[YichengDWu]

What I mean is that one can use Sophon to quickly verify that PINN can solve a certain equation, and I am confident that it is correct. I have not implemented a more fancy adaptive algorithm other than the example given above. I have verified the equations in some adaptive algorithm papers in the past, and Siren + BFGS + finite difference has been able to achieve good accuracy. My personal opinion is that the lack of adaptive algorithm is not a bottleneck affecting the accuracy of NeuralPDE, and it should perform well even without it.

Adaptive sampling algorithm may be a more worthy direction to invest in, it is similar to adaptive weighting but more flexible, only need to sample externally and then remake(prob; p=newdata) instead of embedding the implementation inside NeuralPDE.jl. I have an example in Sophon:
https://yichengdwu.github.io/Sophon.jl/dev/tutorials/Schr%C3%B6dingerEquation/#Customize-Sampling

Or get the residuals for each data point via Sophon.residual_function_1 for custom residual-based sampling.