January 2014 summaries

Linear-Scaling Density Functional Theory applied to optical properties of semiconductor nanocrystals, Nicholas Hine (Cambridge)

The aim of the project was twofold: to study the fundamental properties of semiconductor nanomaterials using linear-scaling Density Functional Theory, and secondly to improve the capabilities of ONETEP for performing this type of simulation. In both respects, the project has been a great success.

Some of the resources were used in the development and benchmarking of a novel scheme for hybrid OpenMP/MPI parallelism, recently implemented in the code by Wilkinson, Hine and Skylaris. This work has led to the publication of a report in J. Chem. Theory. Comput. on novel approaches to parallelism in ONETEP: K. A. Wilkinson, N. D. M. Hine, and C.-K. Skylaris, Hybrid MPI-OpenMP parallelism in the ONETEP linear-scaling electronic structure code: Application to the delamination of cellulose nano-fibrils, J. Chem. Theory Comput. 10, 4782(2014), in which we demonstrate efficient parallelism up to 32768 cores, and strong scaling to the level of 1 core per atom.

A further portion of the resources was used in work by Corsini, in collaboration with Hine, Haynes and Molteni, contributing to two upcoming papers on Ge and CdS nanocrystals respectively, addressing their phase transformations under pressure. The work investigating Ge nanocrystals is a natural follow-up to our earlier work on Si, where we elucidated the mechanism of Pressure-Induced Amorphisation in Si nanocrystals. The new work on Ge involves a new collaboration with experimentalists at Queen Mary University led by Dr Andrei Sapelkin, who have been able to perform a combination of experimental methods including Raman, X-Ray absorption, and transmission light microscopy on Ge nanocrystals within a Diamond Anvil Cell.

Through a combination of experiment and theory we were able to show that the application of pressure drives surface-induced amorphisation leading to Ge–Ge bond over-compression and eventually to a polyamorphic semiconductor-to-metal transformation. We find an intriguing similarity between this phase transformation (observed in both theory and experiment), and recent findings on the low- to high-density liquid transition in Si, suggesting a close connection between liquid metallic Ge and the new high-density amorphous phase we obtain.

Preparing for exascale with mini-applications, Stephen Jarvis (University of Warwick)

This project concerns the benchmarking, performance evaluation and optimisation of three applications: CloverLeaf, CleverLeaf, and TeaLeaf. Each application is a "proxy", developed by academia and industry to understand how key applications will perform at exascale. Our objectives for this work were to:

  • Evaluate the performance of the previously developed versions of CloverLeaf (flat MPI, hybrid MPI/ OpenMP and Co-array Fortran), and the new OpenShmem version, on the new XC30 system to determine which of these programming models exhibits the most promising scalability
  • Evaluate the performance and scalability of the Adaptive Mesh Refinement algorithms in CleverLeaf, and the three linear solver libraries (HYPRE, PETSc and Trilinos) used within TeaLeaf.
  • Investigate different optimisation techniques (such as progress threads and MPI 3.0 constructs) to improve the performance and scalability of Hydrodynamics, Adaptive Mesh Refinement and Linear Solver codes. The ultimate aim is to understand how these codes can be designed to run at exascale
  • Determine whether the technological enhancements incorporated into the Aries NIC on the Cray XC30 which specifically target PGAS-based languages can deliver performance and scalability advantages for PGAS-based approaches compared to their MPI equivalents;
  • Evaluate the performance and scalability of the Cray Aries (Dragonfly) interconnect and assess its suitability to function as a future interconnect for multi-Petaflop and exascale systems;
  • Gather data from these runs on compute/communication breakdown and overlap so that potential bottlenecks at exascale can be anticipated and mitigated
  • Use the outputs from (1)-(6) to inform language, algorithm, and code development choices in our search for an exascale-ready Eulerian structured mesh hydrocode and iterative sparse linear solver software package;
  • Publish this work and make the techniques and codes available to the wider research community.

The research outcomes arising from, this study were:

  • Several optimisations to the implementation of the OpenMP threading constructs within CloverLeaf were examined. These optimisations successfully improved the performance and scalability of the codebase.
  • Several optimisations were also examined and implemented to improve the overall scalability of the application. ARCHER access was crucial in achieving this due to the scale of the computational facilities available on the platform.
  • The evaluation of two PGAS models (CAF and OpenSHMEM) as candidate technologies for implementing structured hydrocodes was also examined. Archer access was again crucial in facilitating this research due to the high performance PGAS implementations available on the platform. The research determined the optimal way to implement applications of this class in these programming models. Additionally it showed that the performance of Message Passing (MPI) and OpenSHMEM is broadly equivalent whilst the performance of the higher-level compiler-based CAF PGAS model is not as yet able to match that of the low-level OpenSHMEM PGAS model. Paper: Experiences at scale with PGAS versions of a Hydrodynamics Application. In Proceedings of the 8th International Conference on Partitioned Global Address Space Programming Models (PGAS2014), Eugene, Oregon, USA, Oct 2014.
  • To examine several techniques and technologies for improving the mapping between application processes and the underlying machine topology. The research showed that utilising these techniques can deliver significant improvements in overall application performance. Paper: Optimising Hydrodynamics applications for the Cray XC30 with the application tool suite. In Proceedings of the Cray User Group 2014 (CUG), Lugano, Switzerland, May 2014. This was paper received the best paper award at the conference.

Undertake an assessment of several automatic application hybridisation technologies. Including Cray Reveal and the DSL (Domain Specific Language) OP2. The research showed that both approaches can automatically generate code which is comparable in terms of performance and scalability to hand-optimised code, for significantly less programming effort. Paper: Performance Analysis of a High-Level Abstraction-based Hydrocode on Future Computing Systems. In proceedings Performance Modelling, Benchmarking and Simulation workshop, Supercomputing 2014.

Large atomistic simulations of a self-assembling peptide nanocage exploring conformational and encapsulation behaviour, Dek Woolfson (University of Bristol)


  1. To extend two simulations of SAGE nanocages begun with early access time on ARCHER (Nov-Dec 2013) leading to a better understanding of the detailed structure and stability of these recently discovered self-assembling peptide systems.
  2. To explore and analyse the diffusion of water and salt through the pores in these cage structures.
  3. To perform a preliminary investigation of the encapsulation of small molecules and proteins inside the SAGEs to determine the stability of the systems under simulation conditions and explore the limits for molecules to pass through the cage pores.
  4. We envisage that these very large atomistic simulations (44 million atoms) will lead directly to a publication in a high ranking scientific journal.
  5. We will use the results of the proposed simulations as the basis of a responsive-mode grant application to the EPSRC to continue the large scale computational modelling of SAGEs to complement the experimental work being performed in the Woolfson and collaborating laboratories.
  6. We expect the synergistic relationship between modelling and experiment to push forward the scientific endeavour around these fascinating and novel peptide-based cage structures, just as we have found in the past with smaller self-assembling peptide structures.


  1. We have successfully extended the SAGE simulations to 50 ns for the case with the dimer linker "N-terminus out" and to 30 ns with the dimer linker "N-terminus in". In both cases the polypeptide network retains its non-covalent interactions and the structures remain intact. The former simulation (dimer-N-out) remains robustly spherical exhibiting low frequency breathing motions consistent with a stable structure, while the latter (dimer-N-in) is broadly similar, the structure is significantly less rigid and floppy, we are working on quantifying this behaviour. These observations are consistent with our previous simulations of patches of the underlying hexagonal peptide network of the SAGEs; this correlation lends confidence to the both molecular dynamics approaches, which will be useful for assessing different aspects of SAGE structure, assembly and dynamics.
  2. Analysis of water and salt diffusion within the data collected is ongoing.
  3. We have successfully run short simulations (10-20 ns) of the small molecules adenosine triphosphate (ATP), carboxyfluorescin (CF) and the protein green fluorescent protein (GFP) inside the more stable SAGE (dimer-N-out). These simulations are robust and the SAGE structure is unaffected by the encapsulated small molecules and GFP. ATP and CF were added to the inner space of the cage at concentrations approaching the experimental solubility limits (100-200) mM and significant aggregation of the small molecules occurred (especially in the case of CF). Nevertheless, we could observe diffusion of individual small molecules through the pores between the peptide fabric of the SAGEs. Longer simulations are required, with lower concentrations of small molecules to quantify this behaviour, but these initial simulations are an excellent proof of principle for such an investment of resources. The encapsulated GFP simulation was also too short to observe interactions with the SAGE, however the protein appears too large to pass through the pores. Again, some self-association of GFP was seen at the high concentration of the protein used (~mM)
  4. The SAGE simulation work on ARCHER form this RAP application, and some further allocation from HECBioSim, has been presented at the MGMS meeting in Oxford this spring and attracted wide interest. For example, one of the principle GROMACS developers Professor Eric Lindahl is eager to adopt the SAGE system as a "real" benchmarking example for large simulations on large HPC machines like ARCHER. Peer reviewed publications will follow.
  5. This work has lead to a successful 18-month ARCHER Leadership Award to extend the SAGE simulations into the microsecond regime and this work will commence in May-June 2015.
  6. The simulation work has led to a further set of experiments in the laboratory exploring how sequence variation in the SAGE components is reflected in SAGE size and peptide orientation.
  7. Woolfson and colleagues are building basic and more-applied collaborations with UK academics (Banting, Bristol; Warren, Kent) and industry (GSK) to explore potential applications of the SAGEs in cell biology and delivery, and as enzyme factories. The detailed simulations being conducted using ARCHER will continue to inform and direct these experimental programmes.

CoI (Richard Sessions) and PDRA (Amaurys Avila Ibarra) were both grateful to attend the ARCHER users training course held in Bristol in January 2014.

DNS of turbulent flow over superhydrophobic surfaces at high Reynolds numbers, Angela Busse (University of Glasgow)

Superhydrophobic surfaces are surfaces with roughness on microscales and a hydrophobic surface chemistry. When they are submerged in water, an air layer or pockets of air can be trapped on these surfaces. One of the applications of superhydrophobic surfaces is hydrodynamic drag reduction: the shear stress at a water-air interface is smaller than at a water-solid interface and thus the trapped air on a superhydrophobic surface can lubricate a flow over the surface.

Drag reduction by superhydrophobic surfaces has been clearly demonstrated for the laminar flow regime. However, in the turbulent flow regime, especially at higher Reynolds numbers, results remain inconclusive: In experiments in some cases drag reduction was observed, whereas in other cases a drag increase was found.

The aim of this project was to investigate turbulent flow over superhydrophobic surfaces at higher Reynolds numbers and the effects of the roughness that is inherent in a superhydrophobic surface on the turbulent flow by conducting direct numerical simulations.

Models for superhydrophic surfaces were designed where part of the roughness structure was exposed above the air layer. Exposed roughness can for example be caused by air-layer depletion which typically occurs for flow at higher Reynolds numbers. A first study showed that at - constant Reynolds number - the drag reduction (compared to flow over a standard smooth no-slip surface) decreased as the exposed roughness height was increased. In a second study the Reynolds number dependence of the flow over a superhydrophobic surface with fixed physical exposed roughness height was investigated. We expected that the observed drag reduction would decrease as the Reynolds number increased due to a stronger adverse effect by the exposed roughness at higher Reynolds numbers. Counter to our expectations the results showed that at higher Reynolds numbers the drag reduction continued to increase significantly. The recorded flow fields are currently investigated in more detail to identify the mechanisms underlying the increase in drag reduction with the Reynolds number and to prepare further flow statistics for a journal paper. The simulations at higher Reynolds numbers would not have been possible without access to ARCHER, since direct numerical simulations at high Reynolds number are very computing intensive.

We expect that the Reynolds number dependence of the drag reduction may show a strong dependence on the shape of the exposed roughness. We intend to extend this research in future by investigating turbulent flow over superhydrophobic surfaces with exposed roughness of different shapes with the aim to find superhydrophobic surfaces with an optimal surface structure for drag reduction.

Structure and electronic properties of buried interfaces in Li-air battery discharge products, Gilberto Teobaldi (University of Liverpool)

The increasing demand for stable, high energy density rechargeable batteries for electronics and long-range electric vehicles motivates the growing interest in developing alternative strategies capable of substituting the existing intercalation Li-ion technologies. Driven by the theoretically up to ten-fold increase in energy-density, great efforts are currently being devoted to the development of Li-air and Li-sulfur batteries, which rests on the availability of stable, industrially scalable, cathodes, electrolyte and metal Li-anodes.

The initial object of this proposal was to obtain preliminary results and insight into the atomic structure and electronic properties of the buried interfaces between cathode products (lithium-peroxide, Li2O2, nanocrystals) in high energy-density Li-air batteries to develop a follow-up grant and an EPSRC Fellowship application for the PDRA involved in this project. The computing resources were awarded from the ARCHER UKCP allocation.

Prompted by unexpected results on the beneficial effects of gas pre-treatment of metal Li-substrates for the stability of Li-anodes obtained within the EU FP7 project SIRBATT, and owing to a change of PDRA in the summer of 2014, the RAP allocation was diverted to explore the opportunities that molecular gas treatment of freshly cut Li-anode could provide for pristine, non-electrolyte based, passivation of Li-anodes.

By interplay between Density Functional Theory geometry optimization and room temperature Molecular Dynamics, we investigated the effects of atmospheric (N2, O2, CO2) and hazardous (F2, SO2) gas decomposition on the relative energy of Li(bcc) (100), (110), and (111) surfaces, their reducing potential (approximated by the work function), and the emerging electronic properties. The simulations suggest that control of molecular-gas induced Li-anode surface reconstructions can be viable, and that substantial changes (up to over 1 eV) in the work function of the passivated system can be achieved by equilibrium gas treatment of freshly cut Li-anodes. A research article based on these results is being submitted in collaboration with experimental SIRBATT partners, and an EPSRC follow up proposal, to be submitted in 2015, is in preparation. Possible involvement of UK experimental group and companies with interests in high energy-density batteries in the EPSRC follow up proposal is being discussed.

Analysis of flow over a tail plane using high order large eddy simulation, Paul Tucker (University of Cambridge)

The objective was to model the type of flow found over an idealized aircraft tail plane configuration and examine the complex interaction of the boundary layer and spanwise pressure gradient generated by the swept control surface. Simulations were performed on an idealized geometrical section with a contoured wall imposing a streamwise pressure gradient. Boundary layer skewness has also been induced with the application of a body-force like spanwise pressure gradient. We successfully achieved the first aspect of studying the flow when there is a streamwise pressure gradient gaining encouraging agreement with benchmark data. Hence, this has set solid foundations. We have managed to extract

the high order statistics necessary to refine RANS models. The case with the superimposed cross flow is still running now under alternative resources. The project proved slightly more challenging than expected since the notional far field boundary condition needed

to be addressed with care and precisely defining the inflow took longer than we expected. However, broadly speaking the project went well. We have now started to consider, using this data, the improvement of the Reynolds stress modelling in the Tau code used by Airbus and are getting ready to make large-scale eddy resolving simulations on actual tail planes. We did not manage to explore the use of synthetic jets being limited by time constraints. We are hoping to present a paper on this successful work at the next AIAA Aerospace Sciences meeting in January 2016.

Sand transport under irregular wave conditions, Ming Li (University of Liverpool)

The transport of coastal sediment are influenced by complex physical processes involving surface wave-induced turbulent oscillatory flows above seabed, coupled multiphase interactions between water and sediment particles, as well as the formation and evolution of bed-forms such as sand ripples. To study these processes over a broad range of space and time-scales, a new numerical model based on a multiphase particle-based Euler-Lagrange (E-L) approach was developed at the University of Liverpool as part of the EPSRC-STW "SINBAD" project. The model treats the sediment phase as a collection of individual particles whose motions are determined by local interactions with the turbulent flow and can therefore be used to study the physical processes important to large-scale sand migration.

RAP access to ARCHER allowed the E-L model to be applied on a massively parallel scale, providing unprecedented detail in the results that can be used to improve the fundamental understandings and engineering design tools. The aim of this RAP project was to apply the E-L model to simulate sediment dynamics above seabed under typical coastal irregular surface waves. Specifically our main objectives were (i) To substantially improve understanding of the near-bed hydrodynamics and sand transport processes occurring under large-scale irregular wave conditions. (ii) To make direct comparisons of net sediment transport rates, suspended sediment concentrations, and sediment-turbulence interaction measures with existing experimental studies of sediment transport in irregular oscillatory wave conditions, as well as new experiments presently being performed by our collaborators in the SINBAD project. (iii) To evaluate the high-risk, high-reward modelling assumptions that have the potential to significantly reduce computational cost associated with simulating the multiphase wave bottom boundary layer using an Euler-Lagrange simulation strategy.

Several important research outcomes have been achieved by this period of ARCHER access. The model was used to simulate sediment dynamics under a variety of conditions corresponding to mild surface irregular waves with rippled bed and more energetic stormy waves with totally flatbed (sheet flows). The results appeared in two conference papers in 2014, and several manuscripts are in preparation for peer review. The main outcomes can be summarized as follows:

  1. The E-L model was systematically validated for simulating coastal sediment transport by making detailed comparisons with the experiments of O'Donaghue and Wright (2004) and Van der werf et al (2007). These simulations in both the ripple bed and sheet flow regimes utilized up to 20M grid points and 15M particles and are the most detailed simulation data of sand transport in surface wave-induced oscillatory flows that we are currently aware of. Comparisons of velocities and concentrations are in very good agreement with the experimental data, allowing us to confidently explore and develop improved scientific understanding from the simulation data.
  2. Simulations of sand ripple creation driven by oscillatory flows due to waves were performed. The results show the creation of small rolling-grain ripples on initially flat bed, ripple merging, and ripple growth to a stable equilibrium state. This type of particle simulation has not been reported previously and is being used to better understand the forces influences sediment particle dynamics during different stages of ripple growth.
  3. The particle based approach naturally accommodates different particle sizes in the simulation - a difficulty for most traditional continuum approach based sediment transport models. As part of a new collaboration with Professor Tom O'Donaghue and Dr. Dominic van der A (U. of Aberdeen), the E-L simulation datasets are being used to explore the particle-size sorting and preferential transport of coarse & fine sand fractions.
  4. Using the Allinea MAP profiling tool a computational bottleneck was identified in the particle collision computations that we addressed using a targeted vectorization strategy. Using a data re-organization and mixed-precision approach, we were able to realize a near order of magnitude speed-up of the related computations through SIMD vectorization of critical loops.
  5. The limitations of using a "coarse graining" strategy, wherein computational parcels are used to represent several individual particles, were explored for particle interactions within oscillatory flows above bed. Results indicate that important physical processes can still be captured using coarser grains. However, simulation results become more sensitive to the fluid phase grid resolution when coarse particles are employed.

New mechanisms in Rayleigh-Benard convection with and without rotation, Lara Silvers (City University London)

This project is aimed at extending our understanding of Rayleigh-Bénard convection. While some experiments of this setup have been done, what could be considered was limited and so to obtain a fuller picture we need to numerically solve the governing equations.

A principal objective for this project was to understand origin and dynamics of large-scale barotropic flows observed in turbulent rotating convection in a Boussinesq fluid. To address this issue required a number of numerical calculations to be carried out using an existing numerical code. After the successful porting and testing of our code to ARCHER that was much quicker than anticipated, production runs commenced. We performed numerical simulations of Rayleigh-Bénard convection using the Boussinesq approximation. In this small project, we have primarily focused on the rapidly-rotating regime where it has been observed using differently constructed models that a large-scale horizontal circulation can be sustained by small-scale turbulent convection.

Key findings of the research were:

  • Large-scale circulation is observed in the Boussinesq limit using the full set of equations valid for any values of the Rossby number.
  • Our research shows that the initial small-scale flow must be sufficiently turbulent for large-scale circulation to present.
  • In addition to being sufficiently turbulent, we showed that the flow must be sufficiently constrained by rotation for large-scale flow to be observed. We quantified this constraint in terms of the Rossby number, which should be of order unity, or lower.
  • We captured where large-scale flow can be observed in a comprehensive diagram for Rayleigh number and Taylor number space, which could not have been achieved without the advanced calculations performed on ARCHER.
  • We were able to show, for the first time, the eventual disappearance of the large-scale circulation at very large Rayleigh number for a fixed Taylor number.
  • Our data enabled us to state that this critical value of the Rayleigh number (above which the vortex mode is not sustained) scales approximately as the Taylor number.
  • We have confirmed, using the full set of equations and a shell-by-shell energy analysis, that the origin of the large-scale flow is due to a non-local inverse cascade of energy and the 2D behaviour of the depth-averaged vortex mode.

While our investigation was comprehensive, we confined the scope of this short project to the incompressible limit in the Boussinesq approximation and focussed principally on the rapidly rotating regime where we noted some interesting, and timely, questions that need to be addressed. There is still more work to be done in this area including comparative analysis between the compressible and Boussinesq modelling results, which is a pressing area of interest that this investigation has highlighted. To address this important topic in this currently highly active research area (see, for example works this year by, Guervilly, C. Hughes, D. W., and Jones, C. A.; Rubio, A. M. Julien, K., Knobloch, E., and Weiss, J. B.) fully we will require numerous large-scale compressible simulations, which is beyond the remit of this project. Hence, we will be required to obtain a much larger allocation via a grant to undertake this timely study in the foreseeable future.

We are happy to report that results from this small project has already been published in a peer-reviewed paper: Inverse cascade and symmetry breaking in rapidly rotating Boussinesq convection, Favier, B., Silvers, L. J., & Proctor, M. R. E. (2014) Physics of Fluids, 26, 9, 096605. Further, has also been presented at conference: Inverse cascade and symmetry breaking in rotating Rayleigh-Benard convection, GdR Turbulence Meeting, June 2014, CEA-PMMH Paris, France. Further oral presentations are intended and further publications may occur.

Hybrid RANS-Large eddy simulation (LES) of a full aircraft wing, Neil Ashton (University of Manchester)

Computational Fluid Dynamics (CFD) has increasingly provided an important design tool for the aerospace industry, used as a supplement to experimental studies. With a desire to reduce noise levels and improve fuel efficiency, reliable CFD simulations of the complex separated turbulent flow around aerospace geometries is becoming an ever more crucial goal. Obtaining accurate estimates for aerodynamic forces is increasingly relevant with more and more focus on reducing aircraft emissions for stricter EU and US regulations. The overall theme of this work is the development and application of advanced CFD models which offer advantages over standard CFD that are currently used in industry. So far these more advanced models have been limited to academic flows due to the computational requirements needed for such models. Fortunately with ARCHER we are able to start to use these methods on large complex geometries.

The main objective of this study was to begin work into the application of advanced hybrid RANS-LES CFD methods (Delayed Detached Eddy Simulation (DDES)) for a full aircraft model. To achieve these two separate but related applications were considered: a full aircraft model and a simplified 3-element airfoil which was of a similar profile to that of a full aircraft wing. In order to understand the flow around a complete aircraft, but at a reduced computational cost, a simpler Reynolds Averaged Navier Stokes (RANS) model was used to compute this case. From this it was possible to compare to experimental data and understand where advanced models may improve the flow physics. Additionally, a 3-element airfoil was computed using these advanced hybrid RANS-LES methods to establish the mesh and numerical scheme requirements for the full aircraft. The results from this 3- element airfoil case was submitted to an AIAA organized aero-acoustics workshop, led by NASA and Boeing. The results compared favourably with both NASA and Boeing and thus now the next objective is to apply the knowledge learnt from this simpler geometry to the full aircraft geometry. Without access to ARCHER we would not have been able to make the progress we have achieved so far.

Transient flow simulation of a Formula 1 model using Nektar++, Spencer Sherwin (Imperial College London)

As engineering design continues to evolve and become ever more complex, there is an increasing demand for more accurate computational flow simulations. With the use of large-scale simulations now becoming more commonplace, industry is increasingly looking to academia to develop the next generation of computational methods for flow simulations, which may provide a mechanism for more accurate results without excessive increases in cost. This desire is driven by current industry-standard tools, which typically rely on time averaged solutions. Whilst this approach is generally computationally efficient, these methods tend to struggle in accurately reproducing the dynamics of flows that are highly unsteady, such as those that involve vortex separation and interaction. In motorsport applications, such as Formula 1, tracking vortices as they form, interact with other vortices and move downstream is highly important in the context of reducing drag and improving track performance. New methods that can accurately resolve such features, while dealing with complex geometry, high Reynolds numbers and scaling to large numbers of processors, are therefore highly desirable.

This project has broken new ground in applying the high-order spectral/hp element method for a proof-of-concept Formula 1 front wing simulation at realistic road conditions. This method has been used in academia for some time, and combines the high accuracy properties of spectral methods with the geometric flexibility of finite element/volume techniques. However, for the most part, challenges in terms of implementation and pre- and post-processing have precluded its use for large-scale, high-Reynolds number flows that are typically of interest to industry. The objectives of the project were therefore spread across two themes: tackling the aforementioned numerical and computational challenges; and, having solved these, investigating large-scale flow physics that is present in this configuration.

Numerical objectives

The starting point of the project was the open-source Nektar++ framework, which encapsulates the core operators of the spectral/hp element method. After working to optimise the strong scaling of the code for which we now see 74% efficiency from 256 to 4096 cores a number of engineering efforts were made to optimise and reduce the per-core memory usage within the code, in order to support the large meshes needed for the simulation and fit within the memory-to-core ratio of ARCHER's compute nodes. Internal input/output formats were also optimised to provide efficient checkpoint and restarting functionality for the very large meshes under consideration. A specialised boundary condition was also written in order to accommodate a rotating wheel connected to the front section. Finally, pre- and post-processing routines were optimised in order to facilitate faster curvilinear mesh generation and parallel post-processing output.

Flow physics objectives

Building upon the engineering work mentioned previously, the main aim of the flow physics objectives focused around obtaining data from proof-of-concept simulations over the front-wing F1 geometry. After working to study the effects of the spectral vanishing viscosity stabilisation scheme that is used to filter out high frequency oscillations between elements, we have been able to generate a number of data sets for further analysis. The figure below shows instantaneous streamlines from one such simulation and the complexity of the resulting flow that we are able to capture with our method.

With more than 200 million degrees of freedom, these results certainly represent the most complex and challenging geometry attempted with these methods to date. They demonstrate that the spectral/hp element method is very well-suited to these types of complex flow simulations and, with our expertise in this area, the results of this project have motivated a follow-on RAP project under the May 2015 call, in which we propose to tackle a more complex configuration that can be more widely compared with existing experimental and computational data.

The use of ARCHER computational resources was crucial in this project’s success. No other UK-based would be capable of providing the necessary computational resources and support required to obtain simulations to such a large degree of detail.

Spectral leading-edge serrations for the reduction of aerofoil-turbulence interaction noise, Jae-Wook Kim (University of Southampton)

The objective of this project is to investigate and understand the physical mechanisms of sound generation and reduction from aerofoils with a wavy leading edge (WLE) situated in a subsonic mean flow that contains freestream turbulence impinging on the leading edge of the aerofoil. It has been discovered that modifying the leading-edge geometry of an aerofoil is effective in controlling and reducing the aerofoil-turbulence interaction (ATI) noise. However, the noise-reduction mechanisms have not been understood in detail.

We carried out high-resolution numerical simulations of the aerofoil-turbulence interaction phenomenon by using our high-order accurate in-house code, CANARD (Compressible Aerodynamics & Aeroacoustics Research coDe) developed at the University of Southampton. The code has been fully validated and efficiently parallelised based on MPICH2 providing a supralinear scalability (at least for up to 2500+ CPU nodes). We chose a sinusoidal profile (along the span) for the modified leading edge geometry and used both flat plate (zero thickness) and NACA aerofoils in the simulations.

The simulation data offered us a few valuable insights into the understanding of the noise-reduction mechanisms. Firstly, the overall sound pressure level (OASPL) that decreased monotonically (linearly) with the amplitude of the WLE was related partially with the sweep-angle effect in the Hill region where the level of surface pressure fluctuations was substantially lower than those at the Peak and the Root of the WLE profile. Secondly, it was found that the noise reduction in the mid- to high-frequency range was contributed by the source cut-off effect taking place mainly in the Hill region due to the geometric obliqueness (sweep-angle effect). The source strength diminished rapidly around the Hill region across all frequencies. The Peak and Root maintained their source strength comparable to that of the SLE (straight leading edge) counterpart. However, it seemed around the Peak region that some of the source power transferred from low frequencies to the high, which might indicate that a nonlinear event took place around the Peak region. A follow-on investigation on this is required particularly from fluid dynamic perspectives. Lastly, the investigation into the phase spectra provided us a significant insight into the understanding of the source relationships, which led to identifying the phase interference effect as one of the mechanisms of ATI noise reduction. The phase interference spectrum at the source (derived in this study) exhibited a noticeable similarity with the noise reduction spectrum at the far field, showing that the local maxima and minima took place at the same/similar frequencies in both the spectra.

The outcomes of the project has produced two conference papers for the 21st AIAA/CEAS Aeroacoustics Conference (at Dallas, Texas, USA in June 2015) and also a journal paper currently under review in Journal of Fluid Mechanics. We are currently undertaking an extended work based on the successful outcomes and discussing with two wind-turbine manufacturers as well as an aeroengine company for practical applications of the WLE geometry.

CFD-based assessment of hydrokinetic and wind turbine power production in real flow conditions, Sergio Campobasso (Lancaster University)

This project has investigated and shed new light on the real operation characteristics of rotary machinery for the generation of tidal and wind power, providing new valuable guidelines for the design of both machine types.

Oscillating wings can extract power from oncoming water or air streams working as hydrokinetic turbines. This device has strong potential and is optimally suited for renewable power generation from river and/or tidal streams in shallow waters. A 1.2 MW commercial prototype is operational since 2014 in the Severn estuary. The power conversion efficiency of this device strongly depends on the fine details of its unsteady hydrodynamics at realistically high Reynolds numbers and in the presence of strong flow three-dimensionality due to finite wing span. Very little work has been carried out in this area. For the first time, the results of the time-dependent computational fluid dynamics (CFD) simulations of this device, performed on ARCHER using the Navier-Stokes research code COSA, have highlighted that using 2D criteria to choose the motion parameters that optimise the power generation efficiency can lead to suboptimal efficiency of the real (3D) installation. A new design guideline for optimizing the hydrodynamic design of real installations has been proposed. The findings of these simulations, presented at the Conference on Modelling Fluid Flows in September 2015 in Budapest, have provided a step change in the research and development of this device, and will accelerate its deployment on a large scale in the UK and abroad.

Geometry errors affecting the rotors of multimegawatt wind turbines are stochastic factors that usually negatively impact wind turbine performance and significantly increase the variability of wind turbine power, annual energy production and, ultimately, wind cost of energy. The relationship between geometry errors and turbine power itself is affected by substantial uncertainty, due to the low-fidelity of the computer codes typically used in industrial design. The ARCHER COSA CFD simulations of wind turbine rotors affected by geometry errors have highlighted a high sensitivity of turbine power to relatively small realistic geometry errors. In collaboration with the Austrian Institute of Technology, simulations of the same rotors with and without geometry errors are being performed using typical low-fidelity industrial tools. The outcome of the joint comparative study of the sensitivities to geometry errors is a significant contribution to the establishment of wind turbine probabilistic design, an area in which Dr. Campobasso and his group are performing worldwide pioneering work.

Non-orthogonality of the wind direction to the wind turbine rotor yields the so-called yawed flow regime, a condition in which wind turbines are expected to experience periodic fluctuations of the generated electrical power and significant periodic fluctuations of the structural loads acting on blade roots and turbine drivetrain, an occurrence that significantly increases fatigue and the possibility of premature structural failure. The low-fidelity analysis and design tools used in most cases for horizontal axis wind turbine development do not enable a reliable estimate of the aforementioned fluctuations. Preliminary analyses of the COSA ARCHER simulations carried out in this project have enabled accurate estimates of such fluctuations, and more strikingly they point to a power reduction due to yawed wind significantly lower than indicated by current engineering estimates. In these analyses, the harmonic balance frequency-domain solver of COSA has been used to perform the required unsteady periodic flow analyses, and cross-comparisons with the time-domain analyses of COSA have shown that yawed wind flows can be predicted by the former solver about 10 times faster than with the conventional time-domain approach and with negligible accuracy penalty. This demonstration has paved the way for using the rapid and accurate harmonic balance frequency-domain CFD technology to include more realism in wind turbine and farm design, such as the impact of shear and thermal stability of the atmospheric boundary layer on the performance of wind power plants.

The simulations performed in this project will form the core part of the PhD project of Mr. J. Drofelnik, one of Dr. Campobasso's research students, who has developed significant HPC skills and fluid mechanics knowledge within this project, the main findings of which will be published in world leading journals by summer 2016.

High Resolution Acoustic Modelling of Aeroengine/Wing Interaction, Sergey Karabasov (QMUL)

Jet-wing interaction noise is a significant contributor to the overall noise signature of modern aircraft. As aeroengine bypass ratios (and consequently fan diameters) continue to increase, the engine must be mounted closer to the wing, and this source becomes increasingly dominant. As such, understanding and modelling it is highly important to aeroengine and airframe manufacturers. By way of illustration, Rolls-Royce and Airbus have spent several million pounds on experimental studies of jet installation effects in the last few years, and this level of funding is expected to continue.

The installation of an aeroengine onto the wing of an aircraft substantially effects that character of the noise it produces. In addition to the reflection in the wing of the sound produced by the jet, the close proximity of the wing trailing edge to the jet mixing layer produces an additional source which can exceed the jet noise at low frequencies. This jet-wing interaction source arises from the diffraction of the jet near hydrodynamic field by the trailing edge, and is loudest in the forward angles. A mechanism has been proposed to explain the observed directivity of this source which involves sound travelling over the upper surface of the wing being diffracted by the leading edge then interfering with sound travelling directly from the trailing edge. This seems to give a reasonable fit to experimental results, at least when there is no forward motion of the aircraft. However, in the case of a moving aircraft, the flow over the wing would be expected to modify the propagation of sound such that it is no longer clear how much, if any, interference would still occur and what the resulting directivity would be. Experimental limitations of open-jet wind tunnels – particularly refraction by the flight-stream shear layer – prevent meaningful measurements at the angles of interest.

The current project has been devoted to a high-fidelity computational modelling of jet/wing interaction based on the in-house Monotonic Large Eddy Simulation CABARET solver coupled with the Ffowcs Williams-Hawkings method for far-field noise calculations with taking the effect of a moving aircraft into account. Results of the computational modelling have been validated in comparison with the co-axial jet/wing experiments conducted at Central Aero Hydrodynamic Institute (TsAGI), Moscow. For a range of operating conditions at the nozzle exit and grid resolutions, the computational results have confirmed a strong amplification of sound at large sound propagation angles to the jet flow for the jet/wing configuration in comparison with the isolate jet case. For small sound propagation angles to the jet, which correspond to peak noise radiation for isolated jets, the effect of the wing on noise is negligible.

The ARCHER allocation had been essential for performing the CABARET calculations at several numerical grid resolutions for the geometry close to the experiment and without assuming any semiempirical viscous boundary conditions on the solid wall surfaces. Preliminary results of the jet/wing interaction modelling were presented at AVIATION 2015 conference. The project has opened up new collaboration opportunities with NASA Glenn Research Center and Aero Acoustics Ltd.