Data analysis in low-code environments

Organizations are increasing their centralization of data activities as data continues to grow. But, the tools landscape is still very fragmented. Business analysts are restricted to spreadsheets and BI tools, for static data manipulation and exploration. Data scientists code predictive models in many languages and rely upon IT for deployment. The data engineers, however, provide everyone with access to data aggregates, which are taken from a variety of data sources on-prem and online.

Low-code environments allow people to work better together, and provide a platform that can be used by many different audiences. Business users can focus their efforts on exploring and aggregating data. Data scientists can use sophisticated machine learning and artificial intelligence (AI), while data engineers can ensure data manipulations are conducted in compliance with company regulations. A low-code environment is essentially a place that allows users to work in a simple environment, while others can use it as a programming environment.

Additionally, together with IT, the team sets up the appropriate productization protocols, so what they created can be continuously deployed into production — as an interactive application, on the edge, or simply automated for regular execution. The compliance department is happy when they have the right environment.

Let’s look at the different stakeholders and how they benefit from a low-code environment.

No code required for business experts

For business analysts to quickly identify trends or save time on regular reporting and auditing, they need to be able to generate summaries and visuals of their data overviews automatically. The ability to see their data from multiple angles can help them gain new insight into ongoing operations.

Low-code environments make these tasks much easier than creating Excel macros. They are also less restrictive than what data aggregation can do within a standard BI program. Business users can create a data flow intuitively and directly without having to touch any code. This “no-code” use case has the added side effect that that process is properly documented and can be explained (or handed over) to others easily.

The door to automated, well-documented data visualisations and aggregations is open to anyone who has the right environment. With more time, our business professionals can explore more data using other techniques. Gradually, they’ll learn more about modern data science and continuously increase their repertoire of methods that help them make sense of their data. A good environment is key to becoming a data scientist. Data science colleagues have already used the same environment and can draw on their experience and help.

Data engineers need low code

One of the greatest obstacles to understanding all the data is being able to generate and share different views quickly. We can still wait for the corporate-wide, well organized, and always up to date. Data warehousesWe can either wait for them to show up or rely on our data engineers who will quickly respond to give us the right view. Both are unlikely to be efficient.

Data experts can create these data views in a low-code environment and then hand them over to their users. Data engineers can create internal data sources that conform to the requirements. Governancerules and their users can use the same environment for further customization of the data view to suit their needs.

Done right, the data engineers can even switch from one data source (e.g., their current cloud storage provider) to another one — or add yet another new source to the mix — without their users needing to worry. Their low-code solutions work flawlessly and they can see the same view of the data. The data engineers continuously update the views in their virtual data warehouse. Again, the low-code environment documents every step along the way.

The data engineers do visual programming of SQL mainly. They can also reach out to provide code snippets if they wish, but this is rare in a low-code environment. If it is, it will be contained in the low-code flow. It will then be managed and documented as the rest.

Data scientists need low code

While there are many data science techniques that can be used to automate model optimization and/or feature design, it is not easy to find new algorithms or implement them in an efficient way. Many environments are either too complex, too simplistic, or do not cover all the details that a data scientist needs. A data scientist wants precise control over all of the little knobs and dials of a learning algorithm, and they want choice — the ability to pick from a wide repertoire of techniques.

Data scientists have flexibility when it comes to the tools they use in a serious low-code environment. It allows data scientists to focus on their work while allowing them to abstract away from interfacing with other versions of libraries and tool interfacing. Data scientists can reach out to code when they wish, but they don’t have to touch the code every time they need to modify an algorithm’s internals. Essentially, this allows visual programming of a data flow process — data science done for real is complex, after all.

The low-code environment can continue to provide access to new technologies and make it future-proof for continued innovation in the field if done correctly. The best low-code environments ensure backward compatibility. They include a way to package and deploy models, as well as all necessary steps for data transformations into the production environment.

There is no code required for the CxO (and any other business users).

The relationship between data science and end users is often strained. The business people often complain that the data folks work slowly, don’t quite understand the real problem and, at the end of it all, don’t quite arrive at the answer the business side was looking for. The data scientists complain about the amount of explaining that they must do and how underappreciated for all their hard work. Both sides are frustrated: the business didn’t get what they wanted, and the data science team doesn’t get the credit.

This is where a low-code environment can prove to be very helpful. It allows the data scientist team to talk with business users about their goals and to show them how to achieve them. The business users will not need to understand all of the nitty-gritty details of how data is blended and which type of ML model is used to make the prediction, but they can understand the flow of the data and provide instant feedback on when they aren’t getting the answers they are seeking. Low-code environments allow for faster turnaround and data flow adjustments are quick and easy.

Data science can no longer be done in isolation. Instead, it is performed collaboratively with both data and business specialists. They can also deploy the API services and web applications quickly in a low-code environment. Instead of creating dozens of different applications, they might choose to just deploy one that provides a little more interaction to address the dozen current needs.

CDO does not require a code

Making sure all data is used properly to help speed up and improve operations everywhere in the organization is still extremely hard, which is why many organizations now have a central “data department.” But that doesn’t fix the problem; it just acknowledges that it exists and puts the responsibility to make things work onto someone’s shoulders.

Low-code environments can eliminate a lot of friction within the organization’s data use. First, data experts can work in a collaborative environment. They don’t need to wait until all data and tools are integrated into one system but can blend data and tools when needed. Secondly, they can — together with the business users — design solutions for the actual end users and can easily and reliably move those solutions into production. A third advantage of a low-code environment is the ability to audit and manage governance.

But looking into the future, there is more: Having built low-code workflows to solve specific problems, their inherent, built-in documentation makes it easy to use them as blueprints for future problems, so the team doesn’t always have to start from scratch. If the team takes the right, modular approach, they can readily build components that solve parts of a problem, such as establishing well-defined access to the organization’s data lakesproviding templates for standard reports. And finally, if it’s an open environment, all of the new technologies that are currently being invented can still be used by the data science team. Adopting such an open low-code environment doesn’t come at the cost of keeping up with the latest and greatest technologies.

A low-code environment makes sense

It is easier to make sense of corporate data in a low-code environment. It enables collaboration between all stakeholders, allows agile creation of new insights, data services and applications and it brings along inherent transparency that’s critical for governance.

Low-code environments are a great way to ensure that everyone within the organization uses modern technologies without switching tools. This is the most critical property: making sure an organization uses data technology that’s truly future-proof and lock-in free.

Michael Berthold is cofounder and CEO at KNIME.