Menu Close

Topic 4

The page is under construction

Topic 4: Federated models and machine learning algorithms coming from different domains than those involved in TERMINET for testing, validating, and demonstrating their performance in the TERMINET use case

Topic Patron: UOWM

Introduction

Nowadays, organisations are in need of implementing smart agents for customers and customising user experiences to upgrade their status and maintain their presence in the market. By applying digital transformation principles, such as big data analytics and predictive maintenance analytics, businesses can effectively optimise their supply chain with insights and predictions. The deployment of Artificial Intelligence (AI) technology in the IoT domain is an effective way for implementing such intelligent services.

TERMINET offers effective synergies of IoT and AI able to create a dynamic environment where companies can gain high quality insights into every piece of data, all the way between what customers are actually looking at until reaching on how employees, suppliers, and partners are interacting with different aspects of the IoT ecosystem. TERMINET contributes to the Next Generation IoT vision by moving decisions closer to the end user, while also considering current operational limitations and strictly adhering to the sensitive data preservation principles. A key technology enabler for this task is Federated Learning (FL). TERMINET adopts the concept of FL to provide efficient distributed AI at the edge by using machine and deep learning algorithms to enable training and inferencing directly on devices like sensors, drones, actuators, and gateways etc. As a result, a) the latency between events is significantly reduced, e.g., a detected signal or the response of a device to the signal, b) there is no need to rely on connectivity anymore and the data can be processed where created or very close to that, c) since the data stay at the edge privacy is ensured, which is complemented by additional privacy methodologies within the Federated system, d) generalization of models is supported since the produced models train on data from multiple sources.

FL is an emerging technology and a new concept to machine learning. It is an enabler that has immense potential to transform a variety of NG-IoT application domains [6]. Being one of TERMINET’s next generation cutting edge technologies, FL technology will be validated and demonstrated in six proof-of-concept, realistic use cases in compelling IoT domains such as the energy, smart buildings, smart farming, healthcare, and manufacturing.

Functional Requirements

The TERMINET project will employ smart solutions having embedded intelligence, connectivity and processing capabilities for edge devices relying on real-time processing at the edge of the IoT network – near the end user. However, the employment of distributed AI and FL faces significant challenges and demands specific requirements to be met, regardless of the application domain [1], [2].  

Regarding federated networks, where various privacy concerns are raised, communication is a critical bottleneck, since it requires data being generated on each device to remain local. It is essential to develop communication-efficient methods that iteratively send small messages or model updates as part of the training process. A process such as this is necessary in order to fit a model to data generated by the devices in the federated network. Another way to reduce communication is to minimise the total number of communication rounds or to decrease the size of transmitted messages in each round.

Another challenge of FL refers to the variability in hardware, network connectivity and power in federated networks. Furthermore, according to the size of the network, as well as the software constraints of each device, only a small number of devices may be active in a specific time period aiming to conserve energy, resulting to the creation of an unreliable communication path. Newly developed FL methods should be able to expect a low amount of participation, tolerate heterogeneous hardware, and be robust to dropped devices in the network.

Statistical heterogeneity is another major challenge of FL networks. This issue refers to the non-identical data distribution across devices, both in terms of modelling the data, and in terms of analysing the convergence behaviour of associated training procedures. A strong federated system should be capable to operate on the global data distribution by utilising personalised or device-specific modelling, regardless of the way the learning process takes place locally in each user’s private data ecosystem.

FL enables data privacy on each device by only sharing model updates instead of the raw data.  Nevertheless, it has been proven that training model updates can also disclose sensitive information, either to a third-party, or to the central server. There have been quite recent attempts towards enhancing the privacy of FL using tools such as secure multiparty computation or differential privacy. However, the employment of such mechanisms that may provide data privacy often result in reduced model performance or system efficiency. It is of paramount importance to understand and balance these trade-offs, both theoretically and empirically, towards realizing private federated learning systems.

Technical Requirements

TERMINET will develop a distributed Federated Learning Framework (FLF), where model training is distributed across a number of edge nodes enabling privacy by design and efficiency in terms of training and processing requirements. Within FLF, the edge nodes use their local data to train advanced machine learning model required by the core AI machine, located at the IoT platform. This architecture allows the data to stay with the user eliminating the need for data transmission and storage to the IoT platform. The edge nodes then send the model updates instead of the raw data to the core AI machine for aggregation.

Figure 1: TERMINET’s FLF four-tier component

In more detail, TERMINET’s FLF is a four-tier component, as presented in Figure 1. In particular, it consists of the FLF Distributed Machine Learning Component (FLF-DMLC) as the first tier, the FLF Distributed Model Optimisation and Synchronisation (FLF-DMOS) as the second tier, the Model Performance Evaluation (FLF-DPE) as the third tier and the FLF Distributed Model Personalisation (FLF-DMP) as the fourth tier. FLF instances are deployed in both the INT-L and in the upper layer, namely the Platform Layer (PLA-L), as presented in Figure 3. This double instance occurs since the distributed architecture is part of FLF-DMLC and the main optimisation and synchronisation components take place both in the edge nodes and the IoT platforms. The FLF-DMLC functionalities in the INT-L include a) decentralized training activities, where the AI models are trained locally at edge nodes using the IoT data streams coming from the IoT infrastructure, b) distributed model personalisation that offers personalization methods that adapt the model for data available on each edge node or end device, individually, and c) functionality to return local models to the global cloud for refining the next-generation global models.

Figure 3: Platform Layer (PLA-L)

To further support and validate the services of TERMINET’s FLF, as well as to enrich the overall TERMINET platform, additional machine learning algorithms and federated models coming from different domains should be implemented and accordingly trained to fit the TERMINET use cases. Among different variants of the federated learning, noteworthy is Federated Transfer Learning (FTL) that allows knowledge to be transferred across domains that do not have many overlapping features and users [3]. In FTL, training with heterogeneous data may present additional challenges, e.g., not all client data distributions may be adequately captured by the model. Furthermore, quality of a particular local data partition may be significantly different from the rest. On the other hand, Vertical Federated Learning (vFL) is another technique that allows multiple parties that own different attributes (e.g. features and labels) of the same data entity (e.g. a person) to jointly train a model [4]. TERMINET use cases can greatly benefit from either newly developed or pre-trained ML models, enhancing the robustness, reliability, and accuracy of TERMINET’s predictive analytics and distributed AI capabilities

References

  1. Li, T. et al. ‘Federated Learning: Challenges, Methods, and Future Directions.’, IEEE Signal Processing Magazine, vol. 37, no. 3, 2020, pp. 50–60.
  2. Q. Xia, W. Ye, Z. Tao, J.Wu and Q.Li, ‘A Survey of Federated Learning for Edge Computing: Research Problems and Solutions’, High-Confidence Computing (2021), doi: https://doi.org/10.1016/j.hcc.2021.100008
  3. S. Saha and T. Ahmad, ‘Federated Transfer Learning: concept and applications’, 2021, https://arxiv.org/abs/2010.15561
  4. J. Sun, X. Yang, Y. Yao, A. Zhang, W. Gao, J. Xie and C. Wang, ‘Vertical Federated Learning without Revealing Intersection Membership’, 2021, https://arxiv.org/abs/2106.05508