From DBA to Data Engineer in Azure

I recently moved a role

From being a DBA Manager, Who is responsible for the operational databases.

I moved to manage the data engineering group.

So what exactly is the difference between the two functions?

DBA - Production Databases:

• SQL\ NoSQL- 24*7, powerful server on premise or on the cloud, managed or semi managed, security tasks, high performance is a target, multiregional, HA as top priority.

• Developers are using Microservices - so we have many applications many services and many many Databases.

• Many kinds of DB's like Cloud IAAS and PAAS.

• Secure and audit the data is must.

• The Clusters must have Uptime as long as we can achive.

• Data Modeling - is so important too.

Challenges and Problems in the data bases systems

1. Lots of DB’s

2. Lots of creators / no standards

3. Lots of Consumers (Query, tools, SLA)

4. Raw data

5. Lots of data resources

6. Data silos

In Data Engineering we have other challenges for example we have Data lake and Data Warehouses :

1. Batch process.

2. Stream Process.

3. many data sources

4. ETL and EL.

5. Data Quality \ Governance \ Catalog.

6. New skill sets

7. Lots of consumers with difference skill sets.

It is a new and fascinating world, with special challenges and its importance is increasing day by day in organizations.

If once only analysts or data scientists consumed information from the Data Lake, today the apps are already taking information from there.

If once it was enough to know SQL to access the information - today you need to know much more and on the other hand the role of the data engineer to make the information accessible to those who need it in the simplest and easiest way.

The interesting challenges will be presented in the following posts.

stay tuned