Evolutionary Excellence: Bridging the Gap in Multilingual AI

Model merging is a hot area in data science that allows developers to combine the capabilities of multiple pre-trained models without the massive computational cost of training from scratch. The students leveraged state-of-the-art parameter merging methodologies such as weighted averaging, task vectors, and TIES/DARE-TIES. They demonstrated a deep understanding of how these algorithms use parameter sparsity, sign control, and interference reduction to resolve conflicts in the weight space when merging models that share similar underlying architectures.

Instead of manually designing model configurations, the team used evolutionary computation to automate the discovery of optimal model structures. They treated layer composition as a search space, encoding each merged model’s structure as a "genome." Using NSGA-II (Non-dominated Sorting Genetic Algorithm II), they performed multi-objective optimization to search over layer source and layer order. This transition shifted their paradigm from merely tuning parameters within a fixed structure to dynamically "optimizing both structure and parameters" simultaneously.

When merging heterogeneous layers from different models, data scientists face "stitching" issues because internal latent representations mismatch in dimension, statistical distribution, and function. To address this, the team engineered a Glue Layer trained with knowledge distillation. They utilized inter-layer and cross-layer distillation losses to align the mismatched distributions, stabilizing the connections across different model boundaries and making the merged network runnable and testable during inference.

Standard machine learning metrics often focus purely on maximizing average accuracy, which can hide poor performance in minor classes or lower-resource languages. The students addressed this by treating cross-language balance as a primary objective. They formulated information-theoretic metrics—specifically semantic entropy (semantic uncertainty) and its cross-language variance—to quantify reliability gaps between languages. By injecting these metrics directly into their multi-objective search algorithm, they successfully optimized for cross-language fairness.

In industrial data science, running deep learning models is tightly constrained by hardware budgets. To manage the immense computational overhead of evolutionary searches and multilingual evaluations, the team designed a "budget-driven" workflow. They built a modularized evaluation pipeline and utilized "fast screening" to programmatically eliminate weak candidate models early. This allowed them to run full evaluations only on top-tier candidates, utilizing structured ablation studies and standardized configuration logging to systematically trace pipeline failures (such as distillation noise or objective design issues).

• Overcoming an initial knowledge deficit

  At the start of the competition, Xianwei was a brand-new PhD student who "knew almost nothing about large language models."

• Adapting to the academic journey

  Supported by senior peers and encouraging advisors, Xianwei embraced the challenge as early-stage PhD practice. By participating actively, Xianwei quickly acquired a foundational understanding of LLMs, successfully overcoming the steep learning curve and laying crucial groundwork for future doctoral research.

• Multi-threaded execution under time pressure

  After slackening off slightly after the first stage, the team faced an incredibly intense second-stage rhythm. To finish their methods, graphical interface, report, and video, they optimized task scheduling and executed tasks in a "multi-threaded manner."

• Fair work division and synergy

  Xianwei highlights that the team "divided the work fairly based on each person's ability" and supported one another. Yu notes that they improved their reverse planning, milestone tracking, and priority control, establishing clear ownership and regular communication to stay aligned during high-pressure periods.

• Budget-driven design under hardware constraints

  Large model training and evolutionary searches are highly expensive and computationally heavy. To work within limited computing resources, the team adopted a "budget-driven design."

• Maximizing utility of existing resources

  They modularized their workflows to prevent repeated evaluation overhead, restricted candidate pools to avoid invalid trials, and ran small, reproducible experiments to screen out weak models early. This clever management of PolyU's computing resources allowed them to run complex architecture searches and multilingual evaluations efficiently.

• Pivoting the merging paradigm

  The team initially relied on standard parameter-level model merging. However, when they realized this method became unstable and caused performance to drop sharply across different languages, they adapted. They pivoted to a joint approach combining parameter merging with an architecture-level search.

• Resolving physical layer mismatches

  When finding that layers from different models did not connect cleanly due to dimension and distribution mismatches, they flexibly engineered a "Glue Layer" trained with knowledge distillation to stabilize cross-model connections.

• Phased workflow and target-setting

  The FLAME competition ran in three stages (intention, mid-term, and final submission). The team strategically set goals at each milestone: ensuring they had a basic working system and reproducible results first, before progressively improving baselines and presentation materials.

• Building a minimal closed loop

  Rather than trying to build a perfect system from scratch, they strategically established a "minimal closed loop" first. They then iteratively refined this base structure step-by-step, using NSGA-II for multi-objective optimization to balance performance and language fairness systematically.