L2RPN is a series of competitions, with the aim of training an Artificial Intelligence agent to operate a power network. The Delft-AI-Energy Lab is hosting a remake of the L2RPN challenge in the Netherlands. The L2RPN Delft 2023 competition offers an exciting opportunity for students and researchers in the Netherlands to test their AI solutions against others in a realistic and challenging environment.
I specialize in combining power system simulation libraries with web technologies. I have experience with power flow simulation, optimization and transient simulation. I also work on solvers for sparse systems of linear equations and sparse matrix ordering algorithms. For web development I typically use Vue with Web Components and have experience with WebAssembly.
Armadillo is particularly notable in that has a Python interface (based on pybind11). Although, sparse matrix support still needs to be added. Also the PyArmadillo API resembles that of MATLAB, which can be convenient when translating.
I am surprised not to see any mention of the OSQP (Operator Splitting Quadratic Program) solver. It is the most impressive open source solver of this type that I have seen published in recent years. It appears to have been developed as a collaboration between Princeton, ETH Zurich, Oxford, Stanford and some other prestigious names. The benchmarks show that it compares favorably with leading proprietary solvers:
The problem described seems to be an ideal use-case for Machine Learning. The MATPOWER Optimal Scheduling Toolkit (MOST) can already solve:
"a stochastic, security-constrained, combined unit-commitment and multiperiod optimal power flow problem with locational contingency and load-following reserves, ramping costs and constraints, deferrable demands, lossy storage resources and uncertain renewable generation."
Much more and it becomes a global optimization problem where you can never really be sure you are not just stuck in a local optimum. The L2RPN (Learning to Run a Power Network) challenge, from RTE-France, is the most interesting effort I have seen applying Machine Learning to energy system management.
Yes to this! However, a while ago I tested OSQP, but I found low precision and the speed didn't justify using it over other solvers. The project remains on my radar due to the strength of the development team -- amazing group -- and knowing that numerical solvers can really shine in one domain while they struggle in others.
Another pair of projects that I find useful are ECOS [1] and CVXPY [2]. CVXPY makes head-to-head comparisons of solvers (including ECOS and OSQP) trivial.
I’m in the optimization space and have never heard of this. Thanks. I recognize many of the names behind it.
I think part of the reason it’s not more popular is because QPs aren’t the most popular problem type. They’re used in MPC problems and any number of L2-loss function problems like ML problems, but these already have custom methods for solving the QP. Doesn’t meant they can’t adopt a standard high performance solver — I think this is very promising.
Also I’m not sure if there’s any political issues that limits them from listing this on COIN OR. That’s usually most folks go to site for discovering new solvers.
Our experience with it on general benchmarking problems is that it can struggle to get sufficiently accurate dual values to the extent that it fails to solve them.
For certain classes of important QP problems, and when optimization to small tolerances is not required, it's undoubtedly a great solver - but it's not a general solver.
Recently, I was able to get Andes v1.3.5 (a Python package for power system transient dynamics simulation using SymPy) to run in WebAssembly using Pyodide v0.16.1.
I'm not your target user, but I'm curious what happens to the pjm5bus_out.txt file that's generated. Can you somehow make that available to JS so it can be downloaded by the user?
I published this just to demonstrate that Time Domain Simulation was working to Hantao Cui, the author of Andes. Small Signal Stability Analysis will require SciPy and an ARPACK package. I would like to create a PWA that runs Andes in a web worker, like I did for MATPOWER:
Andes, a Python package for power system simulation, uses SymPy now. In previous versions, computing the derivatives for dynamic models was done using complex equations written out manually. SymPy greatly simplifies the process of adding custom device models, which power companies often wish to do.
I was able to compile GNU Octave 4.4.1 into Web Assembly for my MATPOWER PWA. When Mozilla was still at full force OctIodide might have been an interesting project for them.
Satisfying the BLAS/LAPACK dependency was one of the biggest challenges. I ended up passing LAPACK 3.4.2 through f2c before compiling the result with emcc. PCRE and some SuiteSparse libraries were the only additional dependencies needed for my project and they compiled with Emscripten without too much difficulty. The rest was just hacking GNU Octave's Autotools build system, which gleans a lot of information from the system environment, into working with emconfigure and emmake. My changes to the GNU Octave source code are available here:
This is very interesting and would never have guessed that it is possible. I would love to read a more detailed write up how all this works (is everything bundled to a large blob for the octave interpreter for example?). Also is the a specific reason you used v4.4.1? And playing with the PWA everything seems instant, but Octave is ~1Gb installed, wouldn't the Octave wasm have to be downloaded to the client (which it doesn't seem to do)?
The GNU Octave interpreter and its dependencies are all in the matpower.wasm file. 19Mb is still quite large for a website, but it is loaded asynchronously and WASM get compiled as it streams in. The interpreter runs in a WebWorker using a Promise based interface so the UI doesn't get held up. Many of the dependencies (e.g. FFTW, CHOLMOD, ARPACK, Qt, HDF5) are disabled as they are not needed to run MATPOWER. There was no particular reason for using v4.4.1. The core functionality of GNU Octave doesn't change much and that version was stable and sufficient for my needs.
Thank you for explaining, I will have to explore this further as this opens up a lot of interesting possibilities. I find it very impressive that you have managed to get virtually no perceptible loading time at all, for octave running on a client (I remember threads now and then about statically compiling octave for portability which all eventually end up with it not being possible so I would never have guessed this was viable).
There are some test cases included in MATPOWER that are around that size and may be imported. I only embedded small test cases in the application to keep the download size down.
I imagine this app being quite useful to lecturers teaching Power Engineering at Universities. Being able to just send students a link and not requiring them to install complex libraries, deal with licenses or restricted trial versions and them not having to use a particular OS or CPU architecture should help get them started quickly.
Thanks for the reply. I'll have to test out the larger models (mainly for curiosity).
Agree that it could be of use for lecturers as most students don't need the full power of PSSE or PowerWorld for the stuff you do in introductory textbooks.
Best of luck in this. I recall during the last FERC Conference on Software that there was at least one academic presentation where they talked about trying to find a good way to visualize these things with a web browser. I can't remember what the specific challenges were though.
Btw...checked out your open source apps. Nice work. I just wish industry was bigger into building their own tools (like was common in the 80s when everyone wrote their own loadflow tools) instead of being reliant upon vendors for everything. There are advantages though to both.
My view is that the open source projects for power system simulation available now are approaching a stage where they can perform all of the main functions of commercial tools. In some cases, they even offer functionality that expensive proprietary applications do not have. However, they are typically provided as libraries or with text-based interfaces.
WebAssembly creates an exciting opportunity to combine these libraries with convenient, reactive user interfaces that help to manage the their complexity. The web development community is so large and the frameworks and tools available are of such high quality that an individual, or small team of developers, can quickly create complex applications that would have previously taken much longer. Ultimately, this should accelerate the process of bringing the latest research to industry.
I've been making a similar argument for awhile. With the libraries now available and high level tools (Python, SQL, Numpy...etc) we can build tiny applications (just a few pages of code) that are easy to maintain and do exactly what we need. So replacing the majority of PSSE isn't very hard. An entire EMS system is another game entirely. It's not just the algorithms, but the full integration with SCADA and other things that most companies don't want to get involved with (too much risk).
On a different note, how are you approaching sparse matrices in Octave? Also, I assumed Octave's performance would be a non-starter here for large models. Is it performant enough for you?
Octave includes support for sparse matrices and MATPOWER makes use of them by default. It doesn't require any additional effort from me. In general, Octave can quite match MATLAB for performance, but it is typically not far off. Running it as WebAssembly also comes with a small performance penalty. I haven't run any benchmarks yet, but this can be expected to improve as browser vendors optimize their runtimes.
Octave is using UMFPACK to solve sparse systems of linear equations. There may be some performance to be gained by using KLU with AMD preordering.
I am interested to know how best to compile BLAS and LAPACK to WebAssembly. Traditionally, implementations optimized for a particular machine architecture are used to extract maximum performance. However, WebAssembly targets a stack-based conceptual machine. At present, I use LAPACK v3.4.2 and convert it to C with f2c before compiling to WebAssembly. It would be interesting to perform some benchmark tests against other implementations and compare across browsers.
Neat and thanks for the explanations! When you say running it as Webassembly, is the entire thing compiled that way, or is that just in the front end with Octave running on the server?
The whole application runs on the client (in the browser). The server just serves up static resources. These are cached by the browser so the application will still work if you are offline. The result is much like the desktop applications you mentioned, but without anything needing to be installed.
The JavaScript application presents the UI and controls editing of a JSON representation of the MATPOWER case structure. This data can be seen using the JSON option on the Export page. The WebAssembly program contains GNU Octave and its dependencies. It is run in a WebWorker (background thread) and the JSON data is passed to and fro. The program that converts the JSON into Octave data structures and back again is available here:
Shouldn't this be a WebAssembly (WASM) only feature? It seems a shame to be introducing the potential for race conditions in JavaScript. Compiling languages that support multiple threads into JS seems to be what is motivating this work.
https://github.com/casecsv
https://github.com/cktcsv