Generative AI is expected to make its mark on every industry in the next decade, as businesses look for ways to improve productivity and enhance customer experience. For data engineering, there are already quite a few use cases being tested by leading-edge companies, with the aim of reducing the amount of manual work engineers need to do and assisting them with code building.
Here are a few use cases where generative AI can help data engineers.
Data cleaning and preparation
Data comes in a wide variety of formats and one of the key factors in a successful data-led project is ensuring that the data is high quality and readable by the end platform or algorithm. For data engineers, there are tools available for reformatting and cleaning data, but these can get stuck at the processing stage due to incomplete data or unsupported formats.
With the natural language processing functionality of generative AI, data engineers will be able to ask for specific cleaning or preparation to be done on a batch of data, avoiding issues where a batch of data has to be scrapped due to it being incompatible.
See also: What’s Changing Faster? Data Pipeline Tech or the Role of the Data Scientist?
Code conversion
During a migration or modernization project, a shift in programming language or platform may require a full code conversion. This is a very time-consuming process, as 1-to-1 changes between coding languages are not always available and programmers need to be able to identify the correct substitute.
As generative AI tools like ChatGPT have been trained on gargantuan amounts of data, it has been considered a natural assistant for programmers, as it is capable of referring to documentation, tested code, and forums to find the optimal conversion between many programming languages.
Generating code
Similar to code conversion, as generative AI tools have been trained on existing code bases and best practices, data engineers can use them to generate new code that aligns with what has already been added. These tools can also analyze existing code and offer recommendations to cut down on the amount of repetitive or boilerplate code.
A step up from this, data engineers can also use these systems to design and implement data pipelines, providing the engineers with more time to analyze data quality and application performance.
See also: MLOps vs DataOps: Will They Eventually Merge?
Testing
Generative AI can be deployed in various forms for testing performance and security. It can generate test cases that fit the profile of the application or service being delivered, including edge cases which may not be thought up by the data engineering team.
Creating visualizations
There are already programs available that take data and visualize it, but with generative AI, data engineers can ask for more niche changes and test out how the data would look in a variety of scenarios. By taking hands off the wheel, data engineers can trial more types of visualizations to find ones that work.
