The Emergence of Operator-neutral Small Cells as a Strong Case for Cloud Computing at the Mobile Edge

Small cells have emerged as a useful tool for supporting increased network capacity through network densiﬁcation but they can also be used to support edge cloud computing services. In this paper we provide a preview of an innovative concept which tackles the consolidation of multi-tenancy in such type communications infrastructures, as well as the placement of network intelligence and applications in the network edge. After surveing the challenges and the enabling technologies, we present the envisaged architecture to manage and control the Cloud-Enabled Small Cell (CESC) infrastructure. Also, at the operation level, we explain the potential advantages of adopting the proposed solutions on the LTE access networks


INTRODUCTION
The 5th generation of mobile networks (5G) demands key features beyond what the current 4G can offer, such as significantly higher wireless capacity, reduced energy consumption per service, and reliable connectivity with very low latency [1].However, to deliver a viable solution meeting all 5G requirements, a substantial change on the mobile network paradigm is inevitable.
Traditionally, to provide coverage in one Point of Presence (PoP), actual installation of physical infrastructure, e.g.Small Cell (SC), is needed.Despite the fact that mounting equipment in one place may not be possible (e.g.dense areas), such an ownership increases operators CAPEX and significantly hampers business agility, particularly when considering the high degree of cell densification needed to deal with the 5G requirements.Moreover, the static nature of physical ownership makes it difficult (impossible in some cases) to handle scenarios with dynamic capacity requirements.For example, a flash crowd event at a venue (e.g., stadium, urban area, etc.) cannot be well-served without over-provisioning of the underlying physical infrastructure.It can be easily translated to more operators expenses (CAPEX and OPEX), which in turn increases the service cost for the end users.To address this issue, the idea of multi-tenancy has been initiated in 3GPP [2] and it is expected to play a vital role in 5G networks.In a multi-tenant scenario, a third party owns the underlying infrastructure and provides access to the actors of the telecom scene like network operators, service providers, Over-the-Top players and so on.Such a sharing increases service dynamicity and reduces the overall cost and the energy consumption.
Furthermore, although nowadays new stakeholders enter the network value chain at an increasing pace, network equipment deployed at the edge and access networks are specialized devices with hard-wired functionalities.Any adaptation to the ever increasing and heterogonous market requirements means a huge investment to change/deploy hardware.Thanks to the advent of cloud computing, Software Defined Networking (SDN) and Network Function Virtualisation (NFV), the idea of having general-purpose computing and storage assets at the edge of mobile networks has emerged.It is a substantial change on the architecture of current Mobile Access Nodes (MAN) (Cloud-Radio Access Network (C-RAN) approach), from being only a wireless head to cloud-enabled device equipped with, e.g., novel processor architectures, Graphics Processing Units (GPU), Digital Signal Processors (DSP), and/or Field-Programmable Gate Arrays (FPGA).In this line, new industry initiatives have already introduced the concept of Mobile-Edge Computing (MEC) [3] and the related key market drivers [4] implementations.
The resulting solution will allow several operators/service providers to engage in new sharing models of both access capacity and edge computing capabilities, i.e., promoting the concept of Small Cells-as-a-Service (SCaaS) based on the conceptual model of network slicing -the logical partitioning of the localized network infrastructure in one PoP.
In this paper, we review the implementation of Cloud-Enabled Small Cells (CESCs), able to support edge cloud computing in a multi-tenant, multi-service ecosystem.To this end, the solution proposed by H2020 5GPPP SESAME project, (see, e.g., Figure 1), is reviewed from different aspects.More specific, Section 2 provides a review of 5G communication challenges.High level architecture as well as multi-tenancy features are explained in Section 3. Enabling technologies are detailed in Section 4, while Section 5 deals with the possible small cell virtualization and function splits.Section 6 focuses on the orchestrator and service function chaining mechanisms.Finally, the paper is concluded by Section 7 which provides a technoeconomic discussion on the impact of SESAME solution.

CHALLENGES FOR 5G COMMUNICATIONS
Today, communication networks are essential means for all areas and sectors of our modern societies and economies, and do constitute critical pillars to assure further evolution and growth.According to the recent market trends and to the actual European policy measures, it is assessed that the communication network and the wider modern services/facilities environment of the year 2020 will be enormously richer and much more complex than that of today.In fact, within the forthcoming years it is expected that the underlying (usually heterogeneous) network infrastructure will be able of connecting everything according to a multiplicity of application-specific requirements (thus including users, things, processes, computing centres, content, knowledge, information, goods), in a purely flexible, mobile, and powerful way.The number of smart terminals, machines, things (with sensors and actuators) attached to current networks is growing exponentially, and soon it will be possible to connect and operate a multiplicity of equipment (including smart home gadgets, vehicles, drones and even robots); this extends our abilities far beyond our current experience of tablet and smartphone connectivity.Such innovative features necessitate and imply for the proper establishment and the operation of a relevant novel infrastructure, able to provide network features and performance characteristics to assure growth.Market actors (such as network operators and service providers, content providers, manufacturers, SMEs, end-users, etc.) are strongly involved in this kind of evolutionary process; this is expected to redefine existing value chains and to reform roles and/or relationships between the players, whilst creating new opportunities.The simultaneous gradual inclusion of modern features (such as of virtualisation and of software-based network functionalities) in communications networks supports the expected transitional process via further strengthening flexibility and reactivity.
The actual European vision towards assessing, understanding and then realizing the wide multiplicity of challenges is to take place via a dedicated Public-Private Partnership (PPP) Programme, able to provide solutions, architectures, technologies and standards for the ubiquitous 5G communication infrastructures of the next decade.More specifically, according to the related European Programme of Reference [5], the following high level Key Performance Indicators (KPIs) have been proposed to frame the research activities until 2020 and are briefly listed as follows: (i) provision of 1000 times higher wireless area capacity and of more varied service capabilities, if compared to those of 2010; (ii) saving up to 90% of energy per service provided.(The main focus in that direction should be in mobile communication networks, where the dominating energy consumption comes from the radio access network); (iii) reduction of the average service creation time cycle from 90 hours to 90 minutes; (iv) creation of a sufficiently secure, reliable and dependable Internet with a zero perceived downtime for services provision; (v) facilitating future very dense deployments of wireless communication links to connect over 7 trillion wireless devices serving over 7 billion people, thus realizing the option of connecting everything or everyone at any time at any place; (vi) enabling advanced user controlled privacy, so that to guarantee a protection level of the facilities offered.
It is expected that the development of the forthcoming 5G systems will be based on an ecosystem of a tight cooperation between industry, SMEs and the research community to develop innovative solutions and to guarantee the acceptance and exploitation of such solutions in global standards and markets, in order to ensure interoperability, economies of scale with affordable cost for system deployment and the end users.The new 5G systems will open new opportunities for efficient services in the business, administrative and private domain.

HIGH LEVEL FRAMEWORK
The key innovations proposed in the SESAME architecture focus on the novel concepts of virtualising Small Cell networks by leveraging the paradigms of a multi-operator (multi-tenancy) enabling framework coupled with an edgebased, virtualised execution environment.The overall, high-level view of the SESAME system is proposed by Figure 2.
To that end, the proposed Cloud-Enabled Small Cell (CESC) offers computing, storage and radio resources.Through virtualization, the CESC cluster can be seen as a cloud of resources which can be sliced to enable multitenancy.Therefore, the CESC cluster becomes a neutral host for mobile Small Cell Network Operators (SCNO) or Virtual SCNO (VSCNO) which want to share IT and network resources at the edge of the mobile network.In addition, cloud-based computation resources are provided through a virtualised execution platform.This execution platform is used to support the required Virtualized Network Functions (VNFs) that implement the different features/capabilities of the Small Cells (and eventually of the core network) and the cognitive management and Self-X operations, as well as the computing support for the mobile edge applications of the end-users.
The CESC clustering enables the achievement of a micro scale virtualised execution infrastructure in the form of a distributed data centre, denominated Light Data Centre (Light DC), enhancing the virtualisation capabilities and process power at the network edge.Network Services (NS) are supported by VNFs hosted in the Light DC (constituted by one or more CESC), leveraging on SDN and NFV functionalities that allow achieving an adequate level of flexibility and scalability at the cloud infrastructure edge.More specifically, VNFs are executed as Virtual Machines (VMs) inside the Light DC, which is provided with a hypervisor (based on KVM) specifically extended to support carrier grade computing and networking performance.
Over the provided virtualised execution environment (Light DC), it is possible to chain different VNFs to meet a requested NS by a tenant (i.e.mobile network operator).Note that, in the context of SESAME, a NS is understood as a collection of VNFs that jointly supports data transmission between User Equipment (UE) and operators Evolved Packet Core (EPC), with the possibility to involve one or several service VNFs in the data path.Therefore, each NS is deployed as a chain of SC VNFs and Service VNFs.
Finally, the CESC Manager (CESCM) is the central service management and orchestration component in the overall architecture figure.Generally speaking, it integrates all the necessary network management elements, traditionally suggested in 3GPP, and the novel recommended functional blocks of NFV MANO [37].A single instance of CESCM is able to operate over several CESC clusters, each constituting a Light DC, through the use of a dedicated Virtual Infrastructure Managers (VIM) per cluster.

ENABLING TECHNOLOGIES
SESAME allows new stakeholders to dynamically enter the network value chain by targeting to the development and demonstration of an innovative architecture, capable of providing SC coverage to multiple operators, as-a-Service.For this purpose the logical partitioning of a LTE Small Cell network to multiple isolated slices, as well as their provisioning to several tenants is envisioned.In our case the consolidation of multi-tenancy and network sharing in LTE infrastructures will be allowed by utilizing the Multi-Operator Core Network (MOCN) protocol, approprately adjusted for the purposes of a small cell network.
In the context of distributed computing on the mobile network edge, small form factor compute nodes, high cores density and low power consumption are essential parameters for consideration.Virtualization is the main technique to assure multi-tenancy and service isolation.Supporting native virtualization extensions, ARM based SoCs fully satisfy those constraints.On the other hand, existing Virtual Infrastructure Managers, such as OpenStack and hypervisors such as the Linux Kernel Virtual Machine (KVM) are already adopted by the industry as de-facto a standard.The convergence point of these technologies is the KVM port to the ARM platform.
Likewise, an accelerated network between VNFs is crucial for the near native performance of the NFVI.A virtual switch (vSwitch) is the key component to ensure interconnection between VNFs and their communication with the outside world.In fact, virtual switches outperform their hardware counterparts in some scenarios, such as VM-to-VM communication.They are also more flexible when it comes to extending their functionalities.Indeed, virtual switches can benefit from different hardware accelerators available on the host, for example cryptochips, embedded hardware bridges, etc.The effect is significant host CPU offloading, allowing to increase VM density.
In the scope of the SESAME project, we implement an accelerated user-space vSwitch solution based on the SnabbSwitch [6] network framework and Open Data Plane (ODP).The virtual switch takes advantage of the LuaJIT (Lua Just-In-Time) compiler dynamic optimizations of the code during execution.ODP, on the other hand, provides API for accessing a big number of hardware and software devices such as NICs and hardware accelerators.The fact that the vSwitch runs entirely in user-space implies there is no performance penalty, otherwise related to context switching.The vSwitch manages VM-to-VM connections, VM-to-NIC attachments as well as commutation between multiple VMs.On the host side VMs expose their NIC as a vhost-user socket, which makes network acceleration possible via zero-copy memory mechanisms.At a higher level the vSwitch is also responsible for ensuring Service Furthermore, the emerging concept of NFV and the expanding offers of VNF coming from both the cloud and telco world, have brought the idea of offering service composition including the VNFs, as a business use case for the network operators in SESAME.As described in the project architecture, the role of the SESAME Orchestrator (in particular the VNF Orchestrator -located within the CESCM) is to manage the deployment of VNFs and network services, over the virtualized infrastructure offered by the Virtual Infrastructure Manager (VIM).Based on the service and the VNFs requirements, the VIM role is to allocate optimally the resources (compute, storage and networking) so as to deploy the service and the VNFs in the shared physical infrastructure.
Depending on the set of available services and VNFs, certain types or templates of VMs will need to be created that will facilitate the service creation and chaining, by allowing their easy and quick configurability and connectivity.These templates are called descriptors and hold detailed information of hardware resources (i.e.storage, switching and computation), VNF descriptors, network service templates and NFV instances.The stored information is used for a wide range of purposes including service graph indication to map incoming edge service requests, identifying the service deployment plan and its interconnection with already existing services.

SMALL CELL VIRTUALIZATION AND FUNCTIONAL SPLITS
In  The emerging trend in small cell virtualization prescribes that BBU functions are virtualised and run in virtual machines (VMs) migrated in a datacentre made of Commercial Off-The-Shelf (COTS) hardware [8].During their lifecycle VNFs can be listed, created, queried, updated, deleted, rebooted, and resised.Not all the baseband processing functions can be efficiently realised in software and, for this reason, purpose-built hardware can be used still.More in general, the functional split divides a small cell in two main blocks: the Physical Small Cell (PSC) and the Virtual Small Cell (VSC) connected by a fronthaul link.The BBU is then connected to an operator's EPC through the backhaul.Depending on the split tight latency constraints for the coordination between VSC and PSC can be hard to achieve in practice.This is one of the hurdles behind RAN virtualisation even in the context of the ETSI NFV Management and Orchestration (MANO) framework [9].
The Small Cell Forum recently published its work on RAN virtualisation [10], in which multiple splitting points between VSC and PSC are evaluated in terms of latency and data rate constraints.Through splitting the RAN functionality in two parts, one executed locally and one executed remotely, it is possible to offer a centralisation gain, while posing challenges on the requirements of the transport network [11].Also splitting enables multiple PSCs being controlled by one VSC, which is a suitable configuration for RAN sharing.Depending on the chosen split, the link requirements are reduced or increased and a different degree of centralisation is achieved.In case the splitting is done at the lowest possible PHY level, the latency requirement can be as low as 250 µs.This imposes to select more challenging and costly technologies such as optical fiber and E-band radio.When the functional split is done at the higher layers, bandwidth and latency constraint are relaxed, while RAN protocol stack dependencies and requirements become more relevant.
Many factors determine the required latency and data rate of each functional split and several options exit for splitting the RAN [10].In overall, split may be realised: (i) at the lower PHY, (ii) at the upper PHY, (iii) at the lower MAC, (iv) at the upper MAC, (v) at the RLC, (vi) at the PDCP and (vii) above the PDCP.As mentioned above, with a split at the lower MAC or below, ensuring time critical coordination between VSC and PSC becomes challenging since scheduling of physical resource Blocks (PRBs) occurs every one millisecond in LTE.Therefore, solutions looking to different separations between VSC and PSC are of great importance to enable a cloud environment which can trade-off costs and complexity but still leveraging on the advantages of NFV.

NFV ORCHESTRATION AND SERVICE FUNCTION CHAINING
As mentioned above, the idea of NFV is to migrate network functions, such as gateways, proxies, firewalls, and transcoders traditionally deployed over specialized hardware (i.e.middle-boxes) to software-based applications, virtual network functions (VNF), implemented and executed over standard high volume servers -in our case Light DC.It provides various benefits, including: i) efficient management of hardware resources, ii) rapid introduction of new functions and services to the market, iii) ease of upgrades and maintenance, iv) exploitation of existing virtualization and cloud management technologies for VNF deployment, v) reduction in CAPEX and OPEX, vi) enabling a more diverse ecosystem and viii) encouraging openness within the ecosystem.NFV Orchestrator (NFVO) in the heart of CESCM is an essential component for deployment, operation, and management/orchestration of network services (NS), especially under the specific 5G reliability and performance requirements.Carrying out such a responsibility means solving a multifold problem with challenging issues.Among them service chaining and service mapping over the disturbed (network of connected CESCs) and heterogonous (CESCs are enriched with different HW accelerators and IT resources) Light DC environment are two fundamental issues to address [12].
A service chaining procedure tries to logically combine different requested NSs, considering their interdependencies.It means that upon receiving a NS request, NFVO has the freedom to chain the functions in the best possible way to fit the requirements of VSCNO while optimizing Light DC resource utilization.Such an optimal resource allocation may include VNF sharing and reuse among one VSCNO running NSs.After the service chaining, the second challenge is to find the best placement for VNFs, considering i) the distributed and heterogeneous nature of Light DC, ii) requirements of the requested NS and iii) its possible impact on the other running NSs.The mentioned problems can be modeled with NP-hard optimization problems, such as Location-Routing Problems (LRP) [13] and Virtual Network Embedding (VNE) [14].Similar to a generic resource allocation problem, a solution can be either based on a complex and time consuming integer liner programming (ILP) formulation or less accurate, light computational heuristic algorithms.Note that depending on the agreed SLAs and SCNO business plan, different optimization objectives (e.g.minimizing energy consumption, maximizing resource utilization, etc.) may be selected.

SDN and traffic steering
To enforce a traffic steering and service chaining, several solutions have been introduced in the academia and among the open source community.Beginning from the OpenDaylight proper chaining solution [15] and few implementations offered form NEC and Ericsson, several standardization groups are also involved in such use case scenarios, one being the OPNFV group [16].Overall, the techniques that deal with enforcing traffic steering on a network level can be divided as: packet based and flow based.The first requires that the packet carry the ID of the chain as encoded information in the header field.This can be done by a simple packet tagging or rewrite mechanisms [17] or by introducing new dedicated protocols, such as the Network Service Header [18] introduced by Cisco and leveraged in the Open Daylight (ODL) community to support their ODL SFC integral project.The problem of rewriting and tagging is that it alters the original datagrams and this can potentially cause a problem for VNFs that require the datagrams in their original structure in order to enforce a decision on the traffic steering, such as the case of virtual deep packet inspection.A typical burden of introducing new headers on the top of the already existing protocols as we mentioned above, is the undesired overhead and complexity agglomerated, as in the case of the additional service header that is introduced in order to enforce endto-end traffic as an overlay connection above the service chain path.
The benefit of SDN in the case of service chain is that employing the Open Flow (OF) principles, the routing can be steered over a specific networking path by programmatically applying OF based rules (flows) on the SDN controller or the virtual switch (OVS) inside the VM hosting the VNF.This involves traffic steering based on flow programming rules by applying actions on the OF ports of the switches, in order to gear the desired route of the packets in the chain.This routing logic in this case is simplified as opposed to the novel header approach, as it leaves the packets untacked and agnostic of the existing chain.Furthermore, it avoids undesired overheads on the top of the IP headers (ex.OpenStack tenant isolation [19]).To have a deterministic routing in the chain a field can be reserved that follows the sequence of the VNF traffic by keeping track of the hops in the chain (the number of interfaces) -the VLAN ID being a suitable candidate as the chain routing does not follow the Ethernet routing.
To support the discussed implementations, a fully SDN enabled network is a prerequisite.This would place the SDN controller as absolute entity in charge of the chain rules over the entire network graph.Considering traffic steering within the Sesame project inside the 5G-PPP ecosystem, requires deeper understanding of the NFV concepts and consolidating the requirements to establish the desired functionality in environment that is not fully prepared to support it.Many efforts have been spent today in creating virtual EPC building blocks.Some virtualization has taken place on the radio access fronthaul side as well, ex.OpenAirInterface [20].SFC concepts based on SDN can be applied in LTE environment to enforce packets among a logical network graph and achieve certain service functionality among the virtualized components such as BSS, HSS, RAN, etc. [21].The SDN controller role has been investigated in the literature along with some experimental implementations in the industry sector [17,22,23,24,25].However bridging the gap between VNF development in a mix cloud and telco ecosystem while offering the ground for a networking solution and traffic steering based on SDN is yet a virgin area and main targeted use case in the SESAME framework.

Composition and service function chaining
From a single data centre point of view, the SDN controller is a component placed on the on the Virtual Infrastructure Manager (VIM).The service orchestrator in Sesame will use the northbound interfaces of the SDN controller application in order to request a composition of network services.As a primary functionality in the case of service chain, the Orchestrator schedules the deployment of the virtual infrastructure, the scheduling of the VNFs and the network service.The dataplane steering and rule enforcement policy are kept within the SDN controller application logic and enforced over the network that hosts the light DC specific NFVs.If the routing involves multiple light DCs, then the SFC approach may alter depending on the placement of the controller within the given architecture.This has to be in accordance with the networking protocols applied in the scenario, for example MPLS and BGP as used today for intra data-centre routing.If SFC is established on GTP tagged traffic that enters the Light DC, then a component/function that removes the GTP header and extracts and reconstructs the IP packets is required as intermediator between the ingress/egress ports of the Light DC.
In order to compose and chain services/functions those need to be described as external dependencies.
How this is to be described is a piece of work currently underway, however there are two identified candidates: TOSCA [26] and NFV-GD [27].With these inputs the NFVO can carry out the composition through orchestration lifecycle.Therefore the composition of a service/function should also maintain the lifecycle as an individual atomic service/function.A Composed Service aggregates/combines services together with orchestration logic.Both Atomic and Composed Services can be used to create further composed services.Included in this process can be the complementary process of service function chaining allowing the chaining of VNFs and services.

TECHNO-ECONOMIC IMPACT AND CONCLUSIONS
Business perspectives of the proposed solution are also under investigation.This is of high importance since new players are expected to enter the market while old interactions, demand models and pricing schemes that seem insufficient and thus, must be modified.Furthermore, in a multi-tenancy and highly heterogeneous environment privacy issues are also significant.A s a next step, revenue flows should be identified.This will be also useful for the techno-economic analysis that will assess the proposed solutions.Towards this direction, the pricing schemes used in SESAME ecosystem (between providers as well as between providers and end-users) need be defined.This is necessary since the softwarization of network along with the as a service concept urge the transition from old-traditional to new pricing schemes.In this case, pricing of pure infrastructure, that is pay a onetime/ up-front fee and receive ongoing connectivity at no incremental cost, is currently of no means.This model should be modified in order to take into account other aspects such as memory or CPU (percentage or number of cores) usage by VMs in a server and / or the time that a function or a service is used.
Of course, it should be metined that in a 5G environment characterized by multi-tenancy, heterogeneity and resource sharing regulatory issues are of high importance and thus should be investigated.This is further enhanced by edge caching functionalities giving the ability to collect and process high volumes of date as well as by the transformation of end users from pure consumers to mixed content consumers and producers.National regulation in terms of privacy, data protection and resource sharing was introduced 20 to 30 years ago and it is thus outdated.The growth of a digital ecosystem based on dramatic changes to technologyand global linked in many countries was not predicted by policy makers or regulators.
Undoubtedtly, the coming wave of 5G innovations will have a concrete exploitation and socio-economic impact by 2020, through the deployment of the so called 5G infrastructure.However, 5G will be much more that the next step beyond 4G, as it is expected to be the core functional system of our modern digital society and economy, thus generating a truly converged and tremendously dense communication infrastructure, integrating IT systems (e.g., processing and storage) with plentiful network resources.So the challenge for 5G is to become a sort of universal, highly flexible and ultra-low latency virtualized infrastructure, capable of serving immense numbers of smart things with significant processing and storage capabilities that may be increased via relevant Cloud-based applications.

(
RLC), Packet Data Convergence Protocol (PDCP) and Radio Resource Control (RRC) layers.In macrocell installations, an eNodeB and its functionalities are split between two main components: the remote radio head (RRH), which amplifies and converts baseband signals to radio frequency (RF) signals, and the baseband unit (BBU) located in an equipment closet.RRH and BBU are connected through an optical fiber to support the Common Public Radio Interface (CPRI) specification constraints [7].