Modern application design with In-memory data fabrics

In this article, author of the book "High Performance in-memory computing with Apache Ignite", discussing the design of modern application architecture with Apache Ignite. Part of this article taken from the book.

Let’s take a quick look at an architecture of a traditional system. The traditional application architecture uses data stores which have synchronous read-write operations. This is useful for data consistency and data durability, but it is very easy to get bottleneck if there are a lot of transactions waiting in the queue. Consider the following traditional architecture as shown below.

High-volume transaction processing.

In-memory data grid adds an additional layer within an environment, which uses the Random-Access Memory (RAM) of the server to store most of all data required by the applications. In-memory data grid sits between the application servers and the data store. In-memory data grid uses a cache of frequently accessed data by the client in the active memory and then can access the persistence store whenever needed and even asynchronously send and receive updates from the persistence store. An application architecture with in-memory data grid is shown below.

By using in-memory data grid, data moves closer to the application endpoints. This approach reduces the response times and can lower transactions times from a few seconds to fractions of a second. This way, application can support extremely large numbers of concurrent transactions involving terabytes of operational data, providing a faster, more reliable transactional experience for customers. It is also a more modern scalable data management system than traditional RDBMS, able to elastically scale as demand increases.

Resilient web acceleration.

With in-memory data grid like Apache Ignite, you can provide fault tolerance of your web application and accelerate your web application performance. Without changing any code, you can share session states between web applications through caches.

This above approach provides the highest level of high availability of a system and a customer experience. Since Ignite is an in-memory solution, the performance of the web session clustering and replication mechanism of user web sessions are very high.

Event processing & real-time analysis.

Data tells the story of what's happing with your business on the background right now. With the IoT as a continuous data source, the opportunities to take advantage of the hot data is greater than ever. Traditional data management system cannot process big data fast enough to notify the business of important events as they occur: such as online credit card fraud detection or risk calculation. Apache Ignite allows processing continuous never-ending streams of data in scalable and fault-tolerant fashion in-memory, rather than analyzing data after it's reached the database.

Not only does this enable you to correlate relationships and detect meaningful patterns from significantly more data that you can process it faster and much more efficiently. Apache Ignite in-memory data grid can manage a tremendous amount of incoming data and push notifications to the business application when changes occur with the server. The Apache Ignite continuous queries capability allows systems to quickly access a large amount of incoming never ending data and take action.

Microservices in distributed fashion.

Microservice architecture has a number of benefits and enforces a level of modularity that in practice is extremely difficult to achieve with a monolithic code base. In-memory data grid like Apache Ignite can provide independent cache nodes to corresponding microservices in the same distributed cluster and gives you a few advantages over traditional approaches.

It allows you to use maximum uses of the data fabrics/grid resources. Services running on the in-memory cluster is much faster than the disk-based application server. Apache Ignite microservice based service grid provides automatically deploy any number of distributed service instance in the cluster.

BigData accelerator.

Hadoop has been widely used for its ability to economically store and analyze large data sets and has long passed the point of being nascent technology. However, it’s batch scheduling overhead and disk-based data storage have made it unsuitable for use in analyzing live, real-time data in the production environment. One of the main factors that limit performance scaling of Hadoop and Map/Reduce is the fact that Hadoop relies on a file system that generates a lot of input/output (I/O) files. An alternative is to store the needed distributed data within the memory. Placing Map/Reduce in- memory with the data it needs eliminates file I/O latency.

Apache Ignite has offered a set of useful components allowing in-memory Hadoop job executing and file system operations. Apache Ignite Hadoop accelerator can automatically deploy all necessary executable programs and libraries for the execution of MapReduce across the JVMs, greatly reducing startup time down to milliseconds. This speeds up analysis by avoiding delays in accessing secondary storage. Also, because the execution engine is integrated with the in-memory data grid, key/value pairs hosted within the data grid can be efficiently read into the execution engine to minimize access time.

Cache as a Service.

Data-driven applications that take too long to load are boring and frustrating to use. Four out of five online users will click away if a page stalls while loading. In-memory data grid can provide a common caching layer across the organization, which can allow multiple applications to access managed in-memory cache.

You can isolate the caching layer from the applications by separating the caching layer from the applications. Any applications (Java, .Net, C++) across the organization can store and read data from the cache. By using in-memory data grid as a service, it’s not necessary to build and deploy local caching infrastructure for each application. Applications can use Apache Ignite as cache-aside or write behind to their data base or load data from the database into the cache. It’s eliminate the complexity of management of the hundred or more separate caching infrastructures.

These are some of the ways in-memory grids like Apache Ignite have served as an essential, architectural component for transforming the way businesses use their data to do business. But that’s not all folks! We will cover a lot of in-memory data grid use cases and application architecture in more depth through the book.