Defining the future direction of IBM z/OS Connect

Having first been released in 2015 z/OS Connect is a well established product with hundreds of customers world wide. In its original form z/OS Connect was released as an Eclipse IDE plugin, with a server runtime on IBM z/OS. For many years the product boasted great success selling to the top banks, retailers, and insurance companies around the world. However, as our customers API strategies continued to mature, so too did the requirements for IBM z/OS Connect. This section of the case study explores the customer pain points and changes in IBM corporate strategy that lead to re-design of IBM z/OS Connect.

Customer feedback has always been at the heart of the decision making process in z/OS Connect. Feedback is constantly gathered from multiple sources including; requests for enhancements (RFEs), customer advocacy calls, design partnership calls, customer workshops, and more. This feedback is typically fed into the future roadmap, and enhancements are delivered in subsequent releases. However, some of these enhancements were deemed unviable due to engineering constraints and left unchanged. This was until they could not be ignored any longer due to popular demand.To better understand these requirements, the design team undertook fact finding research calls with a selection of sponsor users. This research lead to the following conclusions of the major pain points that needed addressing:

Developers desperately need to create APIs that meet company and industry standards. This pain point is associated with three factors;

An important factor in success is the speed and accuracy at which application developers can develop APIs to meet incoming requirements. Research suggests a number of factors that can limit developers agility:

Developers need to create more complex mappings between fields of an API and the back-end system it’s exposing. Use cases for complex mapping vary from simple examples such as combining multiple fields, to highly complex, such as calculating the sum of a field in different currencies.

Devops Engineers struggle to create modern DevOps pipelines for z/OS Connect. This has been attributed to the complexity of the z/OS system, requiring more bespoke knowledge when building a pipelines, and the non-standard artefacts that are created by z/OS Connect. These pipelines can take months to build and DevOps Engineers struggle to find adequate documentation.

Customers whom have purchased multiple IBM integration products struggle to make them work together. An example of this is using IBM API Connect to manage z/OS Connect APIs. While the tools are from the same suite they bare little resemblance. The installation methods differ, the interfaces are dissimilar, and the deployment artefacts are not the same. This disparity leads customers to require more bespoke training and leads to an overall higher maintenance overhead.

Developers needed to run multiple versions of the same API to give consumers adequate time to transition to a new version. This was especially important if a new version was likely to break existing behaviours. In relation to this, developers needed to test new versions of on API and quickly roll back to a previous version in the event of failure.

Today in z/OS Connect the same security rules are applied to every path of an API. Developers needs more granularity in these security controls and the ability to apply different security constraints on different paths of the API.

As with any organisation corporate strategy plays a role in defining the direction of its products. During this period there were two major investments made by IBM that were set to influence the future direction of z/OS Connect, and all other products at IBM.

In 2019 IBM announced its hybrid cloud growth strategy. The objective of hybrid cloud is to bring together public, private, and on-prem infrastructure (such as IBM Z) by building upon common open standards of architecture (containers) to simplify resource management. This strategy is being realised through IBM's monumental acquisition of Red Hat for $34 billion - a market leading Kubernetes platform.

In 2018 IBM released the first iteration of IBM Carbon Design System - an open-source design system with a vibrant community of contributors . This design system marked a huge investment in IBM strategy to offer consistent digital experiences for all its products and offering.

In an effort to formalise our product direction the team gathered in a room for 5 days to undertake a Design Thinking workshop. This workshop bought together team members from around the world with representation from all disciplines. In this workshop we collectively explored the research findings, drew up empathy maps, mapped out a series of as-is scenarios, identified need statements, then discussed IBMs corporate strategy. This helped us establish a shared understanding of the problem space moving forwards.

An important baseline that we needed to establish was our three key personas. User research had highlighted that the responsibilities of one of our personas ‘Alan’ had changed, while another persona ‘Shavon’ (from the previous release) no longer existed in the majority of our users shops.

To conclude the workshop we crafted three Hill statements that would represent our intent moving forward.

These hill statements were later pitched to our sponsor users to validate our directions. After a series of calls, and refinements made to the hill statements the hills were ready to commit to the business.

With conclusion drawn around the future direction of the product, the final stage of the process was to pitch our intentions to the business and our clients. This was achieved by firstly presenting our playback zero to an audience of existing z/OS Connect users, then to the business.

Our playback zero outlined our research findings in the form of an as-is scenario, drawing emphasis to our users pain points. We then went on to pitch a to-be scenario whereby these pain points had been resolved, showcasing out future intentions for the product.

The next stage of development was to explore how our hills translated into a product experience. There was already some ideas how this could be achieved with the technology we had available to us. The team went on to explore these ideas with small proof of concepts and prototypes. The challenge was validating these ideas with our customers to ensure we were moving in the correct direction.

The first intention was to modernise our user interface by shift away from an Eclipse IDE and moving towards a web based experience. While this is a big investment, it was seen to benefit our users in the following ways:

To ensure the APIs looked exactly as our customers required, we would firstly implement for a ‘contract first’ process whereby a pre-defined OpenAPI document was imported into the tooling as a starting point. The back-end interface could then be mapped to this OpenAPI document without compromise. Later releases of z/OS Connect would feature API generation (as existed in the current tooling), and API editing (the ability to edit the API schema in the tooling).

Containerising the z/OS Connect deployment was seen to satisfy many of our customers requirements while also aligning with the corporate direction of IBM. It is however a big step change from todays product and required thorough validation with clients. Adopting a micro-service container approach was seen to offer the following benefits to our customers:

From early discussion with other IBM teams, we learned a technology called JSONata. JSONata was an open source technology designed by IBM which was purpose built for manipulating JSON data. This technology also included a complimentary UI that had been designed by the IBM App Connect team (a product we sought to align with). This was seen to be a perfect fit for our requirements for more complex mapping.

The next stage of the process was to bring these ideas to life, while continuing to collaborate with our sponsor users. The following sections describe how this was achieved: