These two achievements highlight both the ongoing evolution of JuliaSmoothOptimizers (JSO) and its growing impact on large-scale nonlinear optimization problems.

JuMP-dev 2024: Advancing Nonlinear Optimization with JuliaSmoothOptimizers

This year’s JuMP-dev workshop, held independently from JuliaCon for the first time in Montreal, offered a focused platform for deep dives into JuMP and its surrounding tools. In my presentation, I discussed the latest progress within the JuliaSmoothOptimizers (JSO) ecosystem, my slides and the replay.

At the core of my talk was an introduction to new solvers and packages like AdaptiveRegularization.jl, which address the unique challenges of large-scale optimization problems with Adaptive Regularization with Cubics. I emphasized the following key innovations:

• Automatic Differentiation (AD) support and integration with JuMP for easier problem modeling.
• Memory pre-allocation for in-place solvers, reducing runtime overhead.
• Support for multi-precision solvers and GPU-based computations, essential for modern large-scale applications.
• The value of factorization-free solvers, which excel in tackling large, complex problems, such as those in discretized PDE-constrained optimization.

For newcomers to JSO, JSOSuite.jl serves as a critical entry point, simplifying solver selection and benchmarking through automatic algorithm matching. This tool eliminates the complexity of choosing from multiple solvers by providing a user-friendly interface that adapts to the problem at hand. My talk also touched on the broader adoption and longevity of JSO, which now spans over 50 registered packages, making it one of the most comprehensive platforms for numerical optimization.

JuliaCon 2023: JSOSuite.jl – Simplifying Continuous Optimization

While JuMP-dev 2024 focused on recent developments, my publication in The Proceedings of the JuliaCon Conferences looks at the core philosophy and implementation behind JSOSuite.jl. Titled JSOSuite.jl: Solving Continuous Optimization Problems with JuliaSmoothOptimizers, the paper introduces JSOSuite.jl as a package designed to bring ease-of-use to complex optimization challenges.

JSOSuite.jl covers a range of problem types—from unconstrained to generally-constrained and least-squares problems—and eliminates the need for users to understand the intricate details of individual solvers. Instead, the package conducts a preliminary analysis of the problem and automatically selects the most appropriate solver, offering significant advantages to both experienced practitioners and newcomers alike.

This paper builds on the innovations within JSO, reinforcing its versatility and ease of use across various fields and applications. The package is a natural fit for researchers who need efficient, reliable solvers without the overhead of manually configuring them for different problem types.

Looking Forward

Both my presentation at JuMP-dev 2024 and the publication of the JSOSuite.jl paper reflect the significant strides made by the JuliaSmoothOptimizers organization over the past year. The JSO ecosystem is positioned to continue driving innovation in the field of numerical optimization.

I’m excited to see how these advancements will be applied across diverse optimization problems in the coming years and look forward to continuing this journey with the JSO community.

We added a preprint to the HAL open-access repository, A projected dynamical system approach to mineral precipitation-dissolution reactions in geochemistry. We delve into a novel approach to modeling kinetically-controlled precipitation-dissolution geochemical reactions. This paper presents a new model that extends the use of ordinary differential equations (ODEs) to accommodate the complexities of non-smoothness induced by full dissolution and the overlapping nature of reactions and minerals.

The model we developed is a specific type of projected dynamical system. It accounts for the complexities of reactions involving multiple minerals and the participation of a single mineral in several reactions.

Projected dynamical systems are a type of non-smooth differential equation, where the derivative of $x$ follows a dynamic that is projected onto the tangent cone of a convex set (denoted here as $C_0$​). This approach is commonly used to model ODEs that need to satisfy certain constraints. In this particular model, we also incorporate an additional proportionality constraint on the derivative of $x$.

We employed a discretization of the system using an explicit scheme, proving the existence of a solution and the convergence of the approximate solution.

While the code for this paper is not included in the preprint, it is available upon request. This decision was made to ensure that interested readers can access the tools necessary to replicate and build upon our work.

Projected dynamical systems offer exciting new possibilities for modeling complex geochemical reactions. Our work is just one example of how these models can be applied to address long-standing challenges in the field. I look forward to seeing how this approach evolves and hope it inspires further exploration in both research and practical applications.

It was popularized by Elizabeth Dolan and Jorge More in their paper titled “Benchmarking Optimization Software with Performance Profiles” (Mathematical Programming, 2002). The performance profile is a way to summarize and visualize the performance of different algorithms across multiple problem instances. It is very much used in the field of mathematical optimization with nearly 5,000 citations of Dolan & Moré, 2002 listed on Google Scholar at the time of writing.

While the method is often associated with Dolan and More, it’s essential to acknowledge the historical roots of performance ratio usage, dating back to at least 1996 with the work of André L. Tits and Yaguang Yang. This historical context highlights the method’s evolution and widespread adoption over the years.

The primary purpose of a performance profile is to assess and compare the effectiveness of a group of solvers on a defined test collection, utilizing a chosen metric or cost. This assessment is shown in a graph that displays the cumulative distribution function of a performance ratio for each algorithm on every problem instance.

How It Works

While traditional benchmarking efforts often involve extensive tables displaying solver performance on various metrics, they face inherent challenges. The sheer volume of data in tables becomes overwhelming, especially for large test sets, and the interpretation of results from these tables frequently leads to disagreements.

The advantage of using performance ratios lies in their ability to offer insights into the percent improvement of a solver’s metric compared to the best solver, while mitigating the impact of a small subset of problems to dominate the conclusions.

To understand how the performance profile works, let’s break down the process. Benchmarks are generated by running a set of solvers ($S$) on a set of problems ($P$), measuring a chosen metric (e.g., CPU time) $t_{p,s}$ for each solvers $s \in S$ applied to problems $p \in P$. It is essential to note that the selected cost metric must be a positive value, where smaller values indicate better performance.

The performance ratio $r_{p,s}$ of solver $s \in S$ on problem $p \in P$ is defined as the ratio of the solver’s metric to the minimum metric among all solvers for that problem, in other words: \(r_{p,s} := \frac{t_{p,s}}{\min_{s \in S} t_{p,s}}\) where $t_{p,s}$ is the value of the chosen metric of solver $s$ on problem $p$, e.g., the CPU time for solver $s$ to solve the problem $p$. Note that the ratio $r_{p,s}$ is greater or equal to $1$. If the chosen metric is possibly 0 then the denominator should be shifted by a small value.

When the solver fails to solve a problem and triggers an alternative stopping criterion, such as reaching a maximum time or iteration limit, we assign a penalty value, usually $+\infty$.

The performance profile is then defined via the cumulative distribution function for the performance ratio $\rho_s:\mathbb{R} \rightarrow [0,1]$ such as \(\rho_{s}(\tau) := \frac{1}{n_p} \sum_{p \in P} 1_{r_{p,s} \leq \tau},\) where $n_p$ is the number of problems, and $1_X$ is the indicator function.

In words, it means the quantities $\rho_s(\tau)$ represents the percentage of problems that, for solver $s \in S$, a performance ratio $r_{p,s}$ is within a factor $\tau$ of the best possible ratio across all problems.

Note that by definition $\rho_s(\tau)$ is always between 0 and 1 and offers valuable insights:

• The values $\rho_s(1)$ gives the percentage of problems where solver $s$ achieved the best ratio. Note that the sum of these percentages across all solvers may exceed 100% in the case of ties.
• The function $\tau \rightarrow \rho_{s}(\tau)$ is a non-decreasing function.
• All solvers should attain a performance ratio of $1$ for $\tau$ sufficiently large unless some of the problems were not solved. In other words, all the solvers will reach a “plateau” representing the percentage of problems solved.

Advantages

• Comprehensive Comparison: The performance profile offers a comprehensive way to compare the performance of multiple algorithms across a diverse set of instances.
• Visual Representation: The graphical representation of the performance profile is easy to interpret. Researchers can quickly identify which algorithms are more competitive by examining the shape and position of the cumulative distribution functions.
• Instance-Specific Assessment: By considering the performance ratio on a per-instance basis, the performance profile allows to identify algorithms that consistently perform well across a range of problem instances. This is especially important in optimization, where the characteristics of instances can vary widely.
• Robustness Analysis: The profile is useful for assessing the robustness of algorithms. Algorithms that exhibit consistent good performance across a variety of instances will have performance profiles concentrated towards the left, indicating reliability. In Dolan & Moré, 2002, the authors justify that performance profiles are relatively insensitive to changes in results on a small number of problems and that they are also largely unaffected by small changes in results over many problems.
• No Need for Aggregation: Unlike some other performance metrics that require aggregating results over instances, the performance profile considers the performance of algorithms on each instance individually, providing a more nuanced evaluation.
• Instance-Specific Insights: Researchers can gain insights into the types of instances where certain algorithms excel or struggle, helping to identify algorithmic strengths and weaknesses in specific problem scenarios.

Disadvantages

• Dependency on Problem Instances: The performance profile is sensitive to the choice of problem instances. Including a diverse set of instances is crucial, and the results can be influenced by the selection of these instances.
• Metric Sensitivity: The performance profile is based on performance ratios, and the choice of the performance measure can impact the results. Sensitivity to the choice of the objective function or metric is a consideration.
• Interpretation Challenges: While the visual representation is generally intuitive, interpreting performance profiles can be challenging when dealing with a large number of algorithms and instances. Researchers need to carefully analyze the plots to draw meaningful conclusions.
• Dependence on Parameter Settings: The performance of optimization algorithms can be sensitive to parameter settings. The performance profile may not fully capture the impact of parameter tuning, and additional analysis may be needed. Finally, the benchmark should be run with similar stopping criteria for all the solvers.
• Limited to Relative Comparison: The performance profile is primarily designed for comparing algorithms relative to each other. It doesn’t provide an absolute measure of algorithm performance, and researchers may need to complement it with other metrics for a more complete assessment.

Implementations

Several online implementations facilitate the generation of performance profiles for mathematical optimization software. Perprof-py is a notable Python package designed for this purpose, offering compatibility with TikZ and matplotlib for graphical output. Additionally, the Julia programming language boasts two significant contributions in this domain: BenchmarkProfiles.jl and SolverBenchmark.jl. These Julia packages provide effective tools for creating performance profiles and contribute to the growing availability of resources for benchmarking and comparing optimization algorithms. Researchers and practitioners in the field can leverage these implementations to assess the performance of solvers and gain valuable insights into algorithmic behavior across diverse problem instances.

Performance Profile Interpretation

In the realm of performance profiles, Dolan and Moré’s 2002 article provides examples that shed light on the interpretation of performance profiles within extensive benchmarks.

We delve further into the analysis with additional examples aimed at illustrating diverse situations and their nuanced interpretations. Throughout the following examples, we place a particular emphasis on the interpretation of performance profiles, offering readers practical insights into understanding and analyzing results. These examples span various scenarios, showcasing the adaptability of performance profiles in capturing different facets of solver performance across distinct benchmarks.

Additionally, it is essential to note that, in all the forthcoming examples, an unsolved problem is designated with a metric value of $+\infty$.

Example 1: A clear winner

We consider here two solvers on a set of 5 problems.

using BenchmarkProfiles, Plots
T = [
  1.0 5.0;
  1.0 10.0;
  1.0 15.0;
  1.0 20.0;
  1.0 20.0;
]
performance_profile(PlotsBackend(), T, ["Solver A", "Solver B"])

We can observe from this profile that:

• Solver A was better than Solver B on all problems;
• Solver A and Solver B both solved all the problems. Therefore, it is clear that on this test set and according to the chosen metric Solver A is preferable.

Example 2: Fast but not robust

We consider here two solvers on a set of 8 problems.

using BenchmarkProfiles, Plots
T = [
  1.0 5.0;
  1.0 10.0;
  1.0 20.0;
  5.0 10.0;
  7.0 15.0;
  6.0 5.0;
  Inf 20.0;
  Inf 20.0;
]
performance_profile(PlotsBackend(), T, ["Solver A", "Solver B"])

We can observe from this profile that:

• Solver A is better than Solver B on 62% of the problems;
• Solver B solves 75% of the problems within a factor $\approx 2^3$ of the best solver, and all the problems within a factor $\approx 2^5$;
• Solver B solves all the problems, while Solver A only solves 75% of the problems.

Therefore, we can observe a mixed situation here. If one is interested in solving efficiently 75% of the problems, then Solver A is of choice. Howeover, Solver B is more robust as it mangages to solve all the problems.

Example 3: Very small factor

The following exemple shows that we need to pay attention to the factors on the x-axis.

using BenchmarkProfiles, Plots
T = [
  1.0 1.0003;
  1.0 1.0003;
  1.0 1.0003;
  1.0 1.0003;
  1.0 1.0003;
]
performance_profile(PlotsBackend(), T, ["Solver A", "Solver B"])

In this case, it is clear that Solver A is better than Solver B, however the factor of difference is so small that it is difficult to draw a clear line between both. This is a situation where either both solvers are equivalent or the metric chosen is not appropriate.

The difficult case of more than 2 solvers

The following example was taken from Gould & Scott, 2016 where is shown the performance of three solvers on a test set of five problems, where the smaller the statistic, the better the solver performance.

using BenchmarkProfiles, Plots
T = [
  2.0 1.5 1.0;
  1.0 1.2 2.0;
  1.0 4.0 2.0;
  1.0 5.0 20.0;
  2.0 5.0 20.0;
]
performance_profile(PlotsBackend(), T, ["Solver A", "Solver B", "Solver C"])

Solver A is the best on 80% of the problems in the test set, Solver B is not the winner on any. Moreover, Solver A solves all the problems within a factor 2 of the best. Thus, it is the most preferable choice over Solver B and Solver C.

Focusing on Solver B and Solver C: if we are interested in having a solver that can solve at least 60% of the test problems with the greatest efficiency then Solver C seems preferable to Solver B.

Trying to confirm this statement and reproducing the performance profile without Solver A however is showing a different story.

using BenchmarkProfiles, Plots
T = [
  2.0 1.5 1.0;
  1.0 1.2 2.0;
  1.0 4.0 2.0;
  1.0 5.0 20.0;
  2.0 5.0 20.0;
]
performance_profile(PlotsBackend(), T[:,2:3], ["Solver B", "Solver C"])

When comparing two solvers on a given test set, performance profiles give a clear measure of which is the better solver for a selected range. But as the examples above illustrate, if performance profiles are used to compare more than two solvers one cannot directly establish a ranking. The article Gould & Scott, 2016 also contains more realistic examples of such phenomenon.

Profile with more metrics and profile wall

One of the drawback of performance profile is the sensitivity to the chosen metric. A straightforward extension is to consider more than one merit function. The following illustration is selected from the article Migot et al., (2022). DCISolver.jl: A Julia Solver for Nonlinear Optimization using Dynamic Control of Infeasibility. Journal of Open Source Software, 7(70), 3991.

This performance profile compares DCISolver.jl variant LDL against Ipopt on 82 nonlinear continuous optimization problems. It uses two merit functions: the elapsed time to solve a problem, and the number of evaluations of objective and constraint functions. This latter quantity is sometimes used to be independent of the evaluation and the implementation adaptation on the computer, while still being strongly correlated to the elapsed time.

Ipopt solved 72 problems (88%) successfully, which is one more than DCI. The plot on the left shows that Ipopt is the fastest on 28% of the problems, while DCI is the fastest on 72%. The plot on the right shows that Ipopt used fewer evaluations of objective and constraint functions on 70% of the problems, while DCI used fewer evaluations on 24%. Since, it doesn’t add up to 88%, there was a tie in the number of evaluations on 6%.

Overall, this plot tends to show that DCI variant LDL is good implementation as it fast, however it uses more evaluations which can be problematic depending on the modeling tool used to access objective and constraint functions.

It is possible to extend this benchmark to more solvers. The following illustrates a similar benchmark where a third solver is added, KNITRO. The first pair of profile is similar to what has been discussed before. The other profiles are showing two-by-two profiles in order to overcome the difficulty raised earlier.

This type of profile wall allows to build more concrete conclusions on the strenght of each solvers.

More on benchmarks

Performance profiles use a performance ratio instead of relying on the count of function evaluations needed to solve a problem. Consequently, performance profiles do not provide the percentage of problems that can be successfully addressed (within a specified tolerance, $\tau$) based on a particular quantity of function evaluations. This data, which is crucial for users dealing with resource-intensive optimization problems and who are keenly interested in the immediate performance of algorithms, is, however, supplied by the data profiles More & Wild, 2009.

References

Dolan, E. D., & Moré, J. J. (2002). Benchmarking optimization software with performance profiles. Mathematical programming, 91, 201-213. DOI: 10.1007/s101070100263

Gould, N., & Scott, J. (2016). A note on performance profiles for benchmarking software. ACM Transactions on Mathematical Software (TOMS), 43(2), 1-5. DOI: 10.1145/2950048

Moré, J. J., & Wild, S. M. (2009). Benchmarking derivative-free optimization algorithms. SIAM Journal on Optimization, 20(1), 172-191. DOI: 10.1137/080724083

Siqueira, A. S., da Silva, R. G. C., & Santos, L. R. (2016). Perprof-py: A python package for performance profile of mathematical optimization software. Journal of Open Research Software, 4(1), e12-e12. DOI: 10.5334/jors.81

Tits, A. L., & Yang, Y. (1996). Globally convergent algorithms for robust pole assignment by state feedback. IEEE transactions on Automatic Control, 41(10), 1432-1452. DOI: 10.1109/9.539425

This discrepancy posed significant challenges to reproducibility and the continuous improvement of algorithms, ultimately hindering the advancement of knowledge in the field in my opinion. Understanding the reasons behind this gap provide insights into the complexities inherent in code development:

• Providing an open-source implementation means navigating platform choices and addressing questions and issues from users;
• Moreover, the journey from a basic implementation to a robust and efficient one requires a considerable investment of time and effort;
• The process of writing, maintaining and implementing a code wasn’t necessarily valorized in scientific research.

While these challenges could be frustrating, they also presented an opportunity for transformation. The necessity of numerical results underscores the importance of investing the development of codes. By shifting our focus from rewriting similar codes for each publication to refining and enhancing existing implementations, we could pave the way for greater reproducibility and innovation in the field.

Furthermore, the potential impact extended beyond academia. By improving the accessibility and quality of optimization codes, we could empower practitioners to tackle real-world problems with greater confidence and efficacy. This shift toward research-level, super-powerful codes held the promise of addressing some of the most pressing challenges facing our society.

In essence, what initially seemed like a frustrating discrepancy evolved into a call to action. By embracing the challenges and opportunities inherent in code development, we could not only advance knowledge within the field but also make meaningful contributions to real-world problems. This, I believe, is one of the most important challenges in research and one that holds the potential to create lasting positive change.

WHY SOFTWARE MATTERS

Over the past decades, there has been a shift regarding scientific software.

Increased Diversity and Accessibility

Open-source software has helped democratize access to scientific tools and resources, making them more accessible to researchers worldwide. Additionally, the open nature of these projects encourages diversity and inclusivity, as contributors from diverse backgrounds can participate in their development.

Increased Adoption of Open Source

There has been a significant rise in the adoption of open-source software in scientific research. Researchers increasingly recognize the benefits of open-source collaboration, including transparency, reproducibility, and community-driven development.

Personally, my research journey began at INRIA, renowned as one of France’s largest and most dynamic research center in computer science and applied mathematics. At INRIA, the development of software is ingrained in the fabric of their research culture, reflecting a deep commitment to innovation and advancement in scientific computing.

Inria and open source software: unwavering support for nearly 20 years

Growing Emphasis on Reproducibility and Transparency

With the replication crisis in various scientific fields, there’s been a heightened focus on reproducibility and transparency. Open-source software plays a crucial role in enabling researchers to share their code and data, facilitating reproducibility and validation of results.

Towards Reproducibility in Research Software

Best Practices for Computational Science: Software Infrastructure and Environments for Reproducible and Extensible Research

Collaborative Development and Sharing

Platforms like GitHub have facilitated collaborative development and sharing of scientific software. Researchers can openly collaborate, contribute improvements, report issues, and share their work with the wider community, fostering innovation and knowledge exchange.

Hard to believe now but Git exists only since 2005 (A short history of Git) and Github started being developped in 2007 (The untold story of Github). Now, pretty much everyone hosts their open source projects on GitHub, including top companies like Google, Facebook, Twitter, and even Microsoft. Julia programming language is housed on GitHub and it’s entirely public.

In 2023, Github reported 284 millions public repository.

Integration of Best Practices

There’s been a greater emphasis on incorporating best practices in software development within the scientific community. Initiatives like the Software Carpentry and Data Carpentry programs offer training in software development skills tailored for researchers, promoting good coding practices, version control, and reproducible research. While it used to be common to criticize academic code as being poorly organized and in need of extensive refactoring, this perception is becoming less prevalent over time.

Recognition and Reward for Software Contributions

There’s a growing recognition of the importance of software contributions to research outcomes. Funding agencies, institutions, and journals are increasingly valuing and rewarding researchers for their software development efforts, whether through citations, funding, or career advancement.

Even though, I classified this in the new practices, this was long known in Montréal. The story of GENCOL and the company AD OPT is one of Montréal classic using research-level software to solve major challenges

Overall, the past decade has witnessed significant progress in the development and adoption of open-source scientific software. These trends have played a crucial role in advancing scientific research, fostering collaboration, and promoting transparency and reproducibility within the scientific community.

OPEN SOURCE SOFTWARE AND THE ROLE OF JULIA

The past decade has seen the emergence of numerous specialized tools, libraries, and frameworks tailored for scientific computing and data analysis. These resources provide researchers with powerful and efficient tools for tackling complex scientific problems across various disciplines. One that stood out for me is the Julia programming language.

In this context, the plateforme and language plays a major role, and I have grown to think that Julia is a perfect match and particularly in an academic context (but not only). In numerical optimization, and computational science in general, performant open-source codes have been developped using Fortran, C++, and in some measure R and Python. There have been numerous success story, however without bringing together a large community of contributors.

The Julia-verse is mostly focus toward Github, and there is a continuous development toward continuous integration tools that would benefit any Julia package. Thus, making coding and maintaining repository in Julia a better experience.

The Julia programming language version 1.0 appeared in 2018 turning this very promising language into a stable one where large codes can rely on without too much maintenance. Julia has opened numerous success story, one of the early that was impressive was Julia joining the petaflop club Julia Joins Petaflop Club. In other words, Julia is fast, which is good in general because you don’t want to invest time into code that will be obsolete. In my experience, and our recent paper Scalable Adaptive Regularization was another example, Julia is doing more than OK comparing to solvers in Fortran or C++.

Finally, one of the most important point for me is the accessibility of the language. I have been teaching numerical optimization classes for the past years with laboratories in Julia (usually 6 blocks of 3 hours) to students that never coded in Julia, but usually had some Matlab or Python experience. I loved the idea that in a couple of hours students were capable of starting doing meaningful code, and we were able to cover very exciting topics. The same experience extended to the research teams were I have been involved.

To me the perspective of being able to write meaningful and efficient code with few lines, a clear syntax and without 10 years of experience sounds like modern and good science.

Citation File Format (CFF) FOR SOFTWARE

To valorize the coding process in scientific research, the generally accepted procedure recently is to make software citable. Therefore, we would register packages to Zenodo.org to generate a DOI that can be cited, although this is not considered a research paper. Some well-known research journal have a focus on the code either accompanying research or regarding as the main contribution the code itself or some update of it (Mathematical Programming C, INFORMS Journal of Scientific Software, …). A complementary project is to write paper about the code itself and not the associated mathematical content. This is the purpose of journal like Journal of Open Source Software or Journal of Open Research Software. They are peer-reviewed journal where a short article is reviewed as well as the package itself and its documentation. The peer-review process is not anonymous and is done via a Github discussion, which I think is in the open-source spirit we discussed earlier.

What is a CITATION.cff file?

According to [1] CITATION.cff files are plain text files with human- and machine-readable citation information for software (and datasets). Code developers can include them in their repositories to let others know how to correctly cite their software. It is very easy to correctly cite a paper: all the necessary information (metadata) can be found on the title page or the article website. Software and datasets have no title page, the relevant information is often less obvious. People who want to cite your software may ask questions like: What is the name of the software (it’s probably not my_scripts/run.m or analysis.py)? What label should I use to uniquely identify the version of the software I have used? What is the appropriate set of people that should be cited as authors? The person who wants to cite your software will probably not be able to answer these questions accurately and consistently themselves, but you can! Give them all the right information in a CITATION.cff file, and they can cite your software correctly.

Create a CITATION.cff file with ease using the form on the cffinit website.

According to [1], the development of CFF is supported by The Institute for Software Technology of the German Aerospace Center (DLR), The Netherlands eScience Center, and The Software Sustainability Institute.

They choose CFF

Github: When you put a CITATION.cff file in the default branch of your GitHub repository, it is automatically linked from the repository landing page, and the citation information is rendered on the repository page, and also provided as BibTeX snippet which users can simply copy! This makes it easy for other users to cite your software project, using the information you’ve provided. https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/creating-a-repository-on-github/about-citation-files [2] If your repository is hosted on GitHub, they will show the citation information in the sidebar, which makes it easy for visitors to cite your software or dataset correctly.

Zenodo: When you have a CITATION.cff file in your GitHub repository, make a release and publish it on Zenodo via the Zenodo-GitHub integration, Zenodo will use the citation information you’ve provided to populate the publication entry! This makes it easier for software developers and maintainers to publish their software with complete and correct metadata. https://twitter.com/ZENODO_ORG/status/1420357001490706442 [2] When you publish your software on Zenodo via the GitHub-Zenodo integration, they will use the metadata from your CITATION.cff file.

Zotero: When you have a CITATION.cff file in your repository, and someone uses the Zotero browser plugin to import a reference to your repository into their Zotero reference manager, it will use the citation information you’ve provided to populate the reference entry! This makes it easier for users to get a complete and correct reference to your software, that they can use when they cite your software in their work! https://twitter.com/zotero/status/1420515377390530560 [2] People can import the correct reference to your software into the Zotero reference manager via a browser plugin.

Unconstrained Optimization

We consider unconstrained optimization problems of the form \(\underset{x \in \mathbb{R}^n}{\text{minimize}} f(x)\) where $f$ is a twice continuously differentiable function, and its Hessian is Lipschitz continuous. The aim is to study iterative algorithms that converges to a first or second-order stationary points.

A classical approach in this context is to use trust-region method, which is a very robust approach. The Julia package JSOSolvers.jl implements two of such approach, TRON and TRUNK. The book Trust region methods by Conn, Gould, and Toint is a must.

Adaptive Cubic Regularization (ARC)

ARC algorithms, recently explored by Cartis, Gould, and Toint (2011a) and Cartis, Gould, and Toint (2011b) are closely related to trust region methods in that steps are computed by solving a sequence of regularized subproblems. A major theoretical appeal of ARC over TR methods is their optimal worst-case complexity property. Whereas the number of function evaluations required to reach a point $x$ for which $|\nabla f(x)|\le\epsilon$ is $O(\epsilon^{-2})$ for TR, which is no better than steepest descent that number is $O(\epsilon^{-3/2})$ for ARC.

How We Do It?

The standard approach consists in performing aniterative search for the shift akin to solving the secular equation in trust-region methods. Such search requires computing the Cholesky factorization of a tentative shifted Hessian at each iteration, which limits the size of problems that can be reasonably considered. In this article, we propose a scalable implementation of ARC named ARCqK in which we solve a set of shifted systems concurrently by way of an appropriate modification of the Lanczos formulation of the conjugate gradient (CG) method. At each iteration of ARCqK to solve a problem with $n$ variables, a range of $m ≪ n$ shift parameters is selected. The CG variant only requires one Hessian-vector product and one dot product per iteration, independently of $m$. Solves corresponding to inadequate shift parameters are interrupted early. All shifted systems are solved inexactly. Such modest cost makes our implementation scalable and appropriate for large-scale problems.

Numerical Results

The numerical results on CUTEst test problems presented in Dolan and Moré performance profile, with respect to the elapsed time, are comparing:

• ARCqK with the implementation of ARC done in Fortran in the library GALAHAD;
• ARCqK with a classical Steihaug-Toint trust-region method implemented in Julia with the same scheme on problems with $\geq 1000$ variables. More details are given in the article, but overall it shows good prospect for ARCqK as the algorithm is both fast and robust.

Our implementation is designed for large scale problems, so we expect to generalize these results to even larger problems.

Teaching the same course in 2022 provided valuable insights that led to improvements, making the 2023 edition an even more enriching experience. The heart of MTH8408 lies in bridging theory and practice, giving students a well-rounded understanding of optimization. Here, we will take you on a journey through the course, its laboratory sessions, student projects, and the role of the powerful Julia programming language.

Course Overview

MTH8408 is a 36-hour course that equips students with a deep understanding of optimization algorithms, spanning various classes of problems. The primary objective is to ensure that students grasp the core concepts of optimization while gaining hands-on experience. It’s this unique blend of theory and practice that sets the course apart.

One remarkable aspect of the course is the extensive in-class discussions that dissect algorithms, focusing on the intricate details that render them efficient. A prime example is the utilization of matrix-free implementations for some algorithms. Such attention to detail ensures that students not only understand the algorithms but can also implement them effectively.

Laboratory Sessions

In addition to the classroom sessions, students spend 18 hours in the laboratory. Here, they get to grips with the Julia programming language, known for its power and flexibility in the field of scientific computing. More importantly, students have the opportunity to contribute to existing Julia packages, such as OptimizationProblems.jl.

The course uses several Julia packages:

• JuMP.jl/NLPModelsJuMP.jl and/or ADNLPModels.jl: for modeling optimization problems;
• OptimizationProblems.jl: set of test problems;
• JSOSolvers.jl: serve as a model for coding new solvers and is used for comparison with the implemented algorithms;
• Ipopt.jl/NLPModelsIpopt.jl: use Ipopt as a generic solver;
• PDENLPModels.jl: model optimal control problems;
• SolverBenchmark.jl: set of tools to benchmark solvers and generate performance and data profiles.

What’s truly impressive is how quickly students adapt to Julia and these packages. In just a few hours, they are able to code a range of algorithms for unconstrained and equality-constrained optimization problems, as well as linear and nonlinear least squares problems. It’s a testament to both the students’ capabilities and the accessibility of Julia as a programming language.

Student Projects

One of the highlights of MTH8408 is the student projects. These projects allow students to explore areas of interest within the realm of optimization and apply their newfound knowledge and skills. The results from these projects are nothing short of astounding, often reaching the level of numerical results expected in research papers.

Some of the topics that students tackled in 2023 included parameter identification in financial market models, Gaussian process regression, quasi-Newton methods, first-order objective-function-free optimization algorithms, local solutions of the optimal power flow problem, nonsmooth R2 algorithms with constraints, and hybrid methods for nonlinear least squares.

The diversity of these projects demonstrates the breadth of optimization’s applications and the depth of the knowledge acquired during the course. Students are not just learning theory; they are actively contributing to the advancement of optimization research.

Teaching with Julia

Teaching MTH8408 using Julia and its associated packages has been a revelation. The simplicity and efficiency of Julia make it a powerful tool for teaching optimization. However, what truly makes a difference is the usage of a straightforward yet highly effective API, known as NLPModels, along with refined ideas and subsolvers for algorithms.

The students’ ability to implement algorithms and achieve exceptional results is a testament to both their aptitude and the efficacy of these tools. Julia has undoubtedly earned its place in the optimization classroom.

Personal Insights

Teaching MTH8408 is not just a professional endeavor; it’s a source of personal fulfillment. The interaction with students, their enthusiasm, and their achievements are incredibly rewarding. Moreover, teaching this course is a two-way street. It guides my research and helps me understand what truly matters in the world of numerical optimization. The students’ exploration of diverse projects continually expands my horizons and enriches my research endeavors.

Talk Title: Optimization solvers in JuliaSmoothOptimizers

At this year’s conference, I had the honor of presenting a talk titled “Optimization solvers in JuliaSmoothOptimizers”. In this presentation, I had the opportunity to provide an overview of the significant strides made in continuous nonlinear nonconvex optimization solvers by the JuliaSmoothOptimizers (JSO) organization. It has been an exciting journey of growth and innovation, and the conference platform was the ideal place to unveil these remarkable developments.

Abstract: Revolutionizing Optimization with JSOSuite.jl

The heart of my talk revolved around a package called JSOSuite.jl. This new addition to the JuliaSmoothOptimizers ecosystem bridges the gap between users and JSO solvers. JSOSuite.jl eliminates the need for users to be familiar with the nuances of various solvers like DCISolver.jl, FletcherPenaltySolver.jl, Percival.jl, RipQP.jl, and more. This leap in user-friendliness simplifies the optimization process and streamlines benchmarking algorithms. Notably, it paves the way for automatic algorithm selection based on specific problem characteristics.

One of the most promising features of JSOSuite.jl is its ability to facilitate benchmarking, making it easy to assess algorithm performance across different problems. This empowers researchers to make informed choices about the best optimization strategies for their particular scenarios.

Furthermore, I highlighted the performance improvements achieved by the solvers in terms of execution speed and memory utilization. This progress is a testament to the dedication and expertise of the JSO community in refining and enhancing their tools.

JuliaSmoothOptimizers: Forging the Path Forward

From my vantage point, JuliaSmoothOptimizers has solidified its position as a cutting-edge platform for studying numerical optimization and prototyping new algorithms. Over the past three years, the organization has evolved into a state-of-the-art hub for optimization enthusiasts, fostering collaboration and pushing the boundaries of what is achievable in this dynamic field.

As I reflect on my experience at JuliaCon 2023 and the incredible strides made by JuliaSmoothOptimizers, I am filled with a sense of excitement and anticipation for the future of numerical optimization. The innovations unveiled during this conference represent a significant step forward, and I am grateful to have been a part of this transformative journey.

It is accessible in open access here. This post was also published on JSO’s website: juliasmoothoptimizers.github.io.

PDENLPModels.jl is a Julia package that specializes the NLPModel API for modeling and discretizing optimization problems with mixed algebraic and PDE in the constraints.

We consider optimization problems of the form: find functions $y, u$ and $κ \in \mathbb{R}^n$ satisfying \(\begin{array}{lll} \underset{y, u, \theta}{\text{minimize}} \int_\Omega J(y, u, \theta)d\Omega \ \text{ subject to} & e(y, u, \theta) = 0, & \text{(governing PDE on $\Omega$)} \\ & l_{yu} \leq (y, u) \leq u_{yu}, & \text{(functional bound constraints)} \\ & l_{\theta} \leq \theta \leq u_{\theta}, & \text{(bound constraints)} \end{array}\)

The main challenges in modeling such a problem are to be able to discretize the domain and generate corresponding discretizations of the objective and constraints, and their evaluate derivatives with respect to all variables. We use Gridap.jl to define the domain, meshes, function spaces, and finite-element families to approximate unknowns, and to model functionals and sets of PDEs in a weak form. PDENLPModels extends Gridap.jl’s differentiation facilities to also obtain derivatives useful for optimization, i.e., first and second derivatives of the objective and constraint functions with respect to controls and finite-dimensional variables.

After discretization of the domain $\Omega$, the integral, and the derivatives, the resulting problem is a nonlinear optimization problem. PDENLPModels exports the GridapPDENLPModel type, an instance of an AbstractNLPModel, as defined in NLPModels.jl, which provides access to objective and constraint function values, to their first and second derivatives, and to any information that a solver might request from a model. The role of NLPModels.jl is to define an API that users and solvers can rely on. It is the role of other packages to implement facilities that create models compliant with the NLPModels API. We refer to juliasmoothoptimizers.github.io for tutorials on the NLPModel API.

As such, PDENLPModels offers an interface between generic PDE-constrained optimization problems and cutting-edge optimization solvers such as Artelys Knitro via NLPModelsKnitro.jl, Ipopt via NLPModelsIpopt.jl , DCISolver.jl, Percival.jl, and any solver accepting an AbstractNLPModel as input, see JuliaSmoothOptimizers.

Example

The following example shows how to solve a Poisson control problem with Dirichlet boundary conditions using DCISolver.jl: find functions $y \in H^1_0$ and $u \in H^1$ satisfying \(\begin{array}{lll} \underset{y, u}{\text{minimize}} \int_{(-1,1)^2} \frac{1}{2}\|y_d - y\|^2 +\frac{\alpha}{2}\|u\|^2 d\Omega \quad \text{subject to} & \Delta y - u - h = 0, & \text{on } \Omega.\\ & y = 0, & \text{on } \partial\Omega, \end{array}\) for some given functions $y_d:(-1,1)^2 \rightarrow \mathbb{R}$ and $h:(-1,1)^2 \rightarrow \mathbb{R}$, and $\alpha > 0$.

using DCISolver, Gridap, PDENLPModels
# Cartesian discretization of Ω=(-1,1)² in 100² squares.
Ω = (-1, 1, -1, 1)
model = CartesianDiscreteModel(Ω, (100, 100))
fe_y = ReferenceFE(lagrangian, Float64, 2) # Finite-elements for the state
Xpde = TestFESpace(model, fe_y; dirichlet_tags = "boundary")
Ypde = TrialFESpace(Xpde, x -> 0.0) # y is 0 over ∂Ω
fe_u = ReferenceFE(lagrangian, Float64, 1) # Finite-elements for the control
Xcon = TestFESpace(model, fe_u)
Ycon = TrialFESpace(Xcon)
dΩ = Measure(Triangulation(model), 1) # Gridap's integration machinery
# Define the objective function f
yd(x) = -x[1]^2
f(y, u) = ∫(0.5 * (yd - y) * (yd - y) + 0.5 * 1e-2 * u * u) * dΩ
# Define the constraint operator in weak form
h(x) = -sin(7π / 8 * x[1]) * sin(7π / 8 * x[2])
c(y, u, v) = ∫(∇(v) ⊙ ∇(y) - v * u - v * h) * dΩ
# Define an initial guess for the discretized problem
x0 = zeros(num_free_dofs(Ypde) + num_free_dofs(Ycon))
# Build a GridapPDENLPModel, which implements the NLPModel API.
name = "Control elastic membrane"
nlp = GridapPDENLPModel(x0, f, dΩ, Ypde, Ycon, Xpde, Xcon, c, name = name)
dci(nlp, verbose = 1) # solve the problem with DCI

References

Migot, T., Orban D., & Siqueira A. S. PDENLPModels.jl: A NLPModel API for optimization problems with PDE-constraints Journal of Open Source Software 7(80), 4736 (2022). 10.21105/joss.04736

Algorithm

FletcherPenaltySolver.jl is a new Julia implementation of Fletcher’s penalty method, introduced in the ’70s, for equality-constrained nonlinear optimization models

min f(x)     s.t.     c(x) = 0.

where $f:\mathbb{R}^n \rightarrow \mathbb{R}$ and $h:\mathbb{R}^n \rightarrow \mathbb{R}^m$ are twice continuously differentiable. Fletcher’s penalty method is an iterative method that aims to compute a local minimum using first and second-order derivatives.

Fletcher’s penalty method replaces the problem with a parametric bound-constrained optimization problem via Fletcher’s penalty function, whose local minimum can be found by any solver designed for such problems. This function is also smooth under classical assumptions, and the penalty function is exact, i.e. local minimizers are minimizers of the penalty function for all values of the parameters sufficiently large. The main computational kernel for evaluating the penalty function and its derivatives is the solution of a certain saddle-point system. If the system matrix is available explicitly, we can factorize it once and reuse the factors to evaluate the penalty function and its derivatives. The penalty function can also be adapted to be factorization-free by solving the linear system iteratively.

It uses other JuliaSmoothOptimizers packages for development. In particular, NLPModels.jl is used for defining the problem, and SolverCore.jl for the output.

The algorithm is a regularized Fletcher’s penalty algorithm. For equality-constrained problems, the method iteratively solves an unconstrained problem. Any solver compatible with Stopping.jl can be used. By default, we use ipopt from NLPModelsIpopt.jl to solve the subproblem, but other solvers can be used such as knitro from NLPModelsKnitro.jl or any solvers from JSOSolvers.jl. The Stopping version of these solvers is available in StoppingInterface.jl. If you have a solver that handles equality-constrained, it is also possible to solve linear equality constraints separately.

It uses LDLFactorizations.jl by default to evaluate the derivatives of the penalized subproblem, but one can also use a matrix-free version with Krylov.jl.

Installation

pkg> add FletcherPenaltySolver

Example

using FletcherPenaltySolver, ADNLPModels

# Rosenbrock
nlp = ADNLPModel(x -> 100 * (x[2] - x[1]^2)^2 + (x[1] - 1)^2, [-1.2; 1.0])
stats = fps_solve(nlp)

# Constrained
nlp = ADNLPModel(
  x -> 100 * (x[2] - x[1]^2)^2 + (x[1] - 1)^2,
  [-1.2; 1.0],
  x->[x[1] * x[2] - 1],
  [0.0],[1.0],
)
stats = fps_solve(nlp)

References

Estrin, R., Friedlander, M. P., Orban, D., & Saunders, M. A. (2020). Implementing a smooth exact penalty function for equality-constrained nonlinear optimization. SIAM Journal on Scientific Computing, 42(3), A1809-A1835. 10.1137/19M1238265

How to Cite

If you use FletcherPenaltySolver.jl in your work, please cite using the format given in CITATION.cff.

Dominique Orban gave a complete overview in 25 minutes (!!) of the JuliaSmoothOptimizers organization. The talk mentions all the key packages to model optimization problems (NLPModels.jl & Co), solve models (JSOSolvers.jl & Co) and do simple yet amazing benchmarks with SolverBenchmark.jl.

Geoffroy Leconte presented RipQP.jl a multi-precision algorithm for convex quadratic optimization. The multi-precision feature illustrates the strength of pure Julia implementations, and ensuing performance.

You can re-watch the three talks on Youtube:

• A multi-precision algorithm for convex quadratic optimization
• The JuliaSmoothOptimizers (JSO) Organization.
• PDE-constrained optimization using JuliaSmoothOptimizers.